AI Rework Happens Before Deployment, Where DORA Metrics Cannot See It

Most AI rework happens before code is deployed, and the main delivery metric for rework only counts what happens after. Michael Hill reports in a recent article that AI rework costs engineers at least 139 hours a year, and this piece follows one of his recommendations, tracking that rework as its own category, to the question of how a team would actually measure it.

Hill reports that AI speed gains are being paid back in rework other teams absorb

Hill’s article covers a new ActiveState survey of more than 300 security, governance, platform, and engineering leaders at organizations with over 1,000 employees in North America and Western Europe. In it, 43% say at least 10% of an engineer’s time goes on fixing or revisiting AI-assisted code. Taking every respondent at the bottom of their range, the report arrives at a floor of 139 hours per engineer a year, which Hill notes is roughly the capacity of seven engineers in a 100-person organization. The survey also finds that half of organizations let new dependencies reach production without review, and that only 31% have formally quantified the cost of unmanaged open-source risk.

Hill argues that the gains show up in velocity metrics while the rework hides in sprint carryover and unplanned tickets, so “Leadership sees the speed and rarely the full price.” Among his recommendations is to track rework on AI-assisted work as its own category.

DORA’s rework rate only counts AI rework that reaches production

The obvious place to look for a rework number is the DORA framework, which Hill’s article already cites for its finding that AI adoption came with lower delivery throughput and stability. DORA now describes instability with two metrics. Its metrics guide defines change fail rate as the ratio of deployments that need immediate intervention, such as a rollback or hotfix, and deployment rework rate as the ratio of deployments that are unplanned and happen as a result of an incident in production.

Both definitions start at the deployment. The rework Hill describes, an engineer revisiting AI-assisted code during review or carrying a half-fixed ticket into the next sprint, never becomes an unplanned deployment. It is absorbed before release. A team could lose the 139 hours the survey estimates and still report a clean deployment rework rate, because the metric was designed to measure production instability, and it does that job well. Pre-release rework is a different quantity, and it needs its own instrument.

Self-reported AI rework estimates are unreliable in both directions

The survey number has a limit of its own. It comes from leaders estimating a share of engineers’ time, and there is good evidence that people misjudge how AI changes their work. In July 2025 Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein of METR published a randomized controlled trial in which 16 experienced open-source developers completed 246 tasks in projects they knew well, with AI tools randomly allowed or disallowed. Before starting, the developers forecast that AI would cut completion time by 24%. Afterwards they estimated it had cut it by 20%. Measured completion time actually rose by 19%.

That trial measured overall task time with early 2025 tools, so it does not tell us how much of the slowdown was rework. What it shows is that perceived and measured effects can differ in sign, and the people closest to the work did not notice. That cuts against any estimate built on recollection, including the 139-hour floor. The true figure for a given team could be higher or lower. Hill’s point that only 31% of organizations have formally quantified their open-source risk applies here too: an informal estimate of AI rework is a guess.

AI rework can be measured from systems a team already runs

If the cost lives before deployment and people cannot estimate it, the measurement has to come from the delivery systems themselves. Three signals are already recorded by most teams. The first is pull request revision after review: the number of follow-up commits and review rounds a change needs before it merges. The second is short-horizon rewrite: lines changed or deleted within a few weeks of merging. The third is ticket reopening and carryover: work marked done that comes back, and sprint items that roll forward, which is exactly where Hill says the rework hides.

None of these signals needs a person to estimate anything. Each one can be compared for changes made with heavy AI assistance and changes made without it, or for the same team before and after a tool rollout. Read alongside DORA’s deployment rework rate, they give a leader both halves of the picture: the rework that is caught before release and the rework that escapes into production.

Labeling work as AI-assisted gets harder as adoption grows

There is a boundary on this approach. Hill’s recommendation depends on knowing which work was AI-assisted, and that line is blurring. When nearly every change touches an assistant at some point, a binary label stops separating anything. At that stage the useful comparison shifts from AI versus non-AI work to the team’s own trend over time, and to rework by type of change, such as dependency additions, where the survey finds review most often happens after the fact.

There is also a counter-case on incentives. If rework becomes a target, engineers can squash commits, avoid reopening tickets, and file fixes as new work. The number is most useful as a diagnostic that tells leaders where to invest, for example in the faster approved dependency path Hill recommends, and it loses that value once people are judged by it.

Why This Matters for Engineering Leadership

Engineering leaders defend AI tooling budgets with numbers, and the numbers they usually have are velocity and DORA metrics. Hill’s article shows that the cost side of AI-assisted development lands in places those metrics were not built to watch. DORA’s own definitions confirm that its rework metric begins at deployment, and the METR trial shows that asking people for an estimate can produce the wrong sign. A leader who adds pre-release rework signals to the dashboard, and keeps them out of individual targets, can put a measured cost next to the measured speed. That is the comparison a board or a CFO will eventually ask for, and it is also the evidence needed to justify upstream controls before rework turns into an incident.

FAQ