To learn whether AI coding tools can lift velocity on hard work, a team has to write down its expected time before the task starts and measure the real time after it ends, because developers in a randomized trial believed AI had made them faster when it had made them slower. Arjun Garg argues in a recent article that flat velocity does not by itself show the tools have failed, and this piece follows the test he proposes to one question: how would a team know what the test found?
Garg argues flat velocity can come from habits as much as from tools
Garg separates two explanations for AI underdelivering on roadmap-critical work. A capability gap means the tool cannot do that class of work reliably with the context and review a team can give it. A habit gap means the team stops shaping important work so AI can contribute and falls back on asking for another senior engineer. He reports lining up AI credit spend against delivery signals in his own organization: the heaviest-spend group spent roughly twice what the middle group did and did not produce a higher median number of merged pull requests. He declines to call that waste, because pull request count measures volume and heavy users may be working on harder problems.
His proposal is to change the default question. Once a human owns the design, the risks and the acceptance criteria for a hard but bounded task, someone asks what an agent could attempt before the work is priced as senior engineer time. If the agent carries more than expected, that points to a habit gap. If it stalls with clear design, clean context and a fair review loop, that points to a capability gap. He warns against turning the question into a dashboard metric: “Once it becomes a number, it becomes something to game.”
Teams cannot read an AI coding trial from how the work felt
Garg’s three outcomes (the agent carried more than expected, the agent stalled, the result was messy but salvageable) are judgments a team makes after the work is done. The strongest published evidence on AI coding speed says that this kind of judgment is unreliable, even for experienced engineers working on code they know well.
In July 2025, METR published a randomized controlled trial by Joel Becker, Nate Rush, Elizabeth Barnes and colleagues. Sixteen experienced open-source developers completed 246 tasks in mature projects where they had, on average, five years of prior experience. Each task was randomly assigned to allow or disallow AI tools, mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet. Before starting, the developers forecast that AI would cut their completion time by 24%. After finishing, they estimated it had cut their time by 20%. The measured result was that allowing AI increased completion time by 19%. Experts in economics and machine learning had predicted reductions of 39% and 38%.
A team that runs Garg’s test and then asks whether the agent helped will get the same kind of answer METR’s participants gave. The work will feel faster or slower, and the feeling may point the wrong way. The spend data Garg examined has a sibling problem: it measures activity. A post-task impression measures sentiment. Neither one measures elapsed time against an expectation set in advance.
A forecast written before the task turns the AI coding trial into evidence
The fix is cheap, because teams already produce most of the data. Before the agent attempts the implementation, the design owner records two numbers: the estimate for doing the work without the agent, which the team needs anyway to price it as senior engineer time, and the share of the implementation they expect the agent to carry. After the work merges and meets the acceptance criteria the human set, the team records elapsed time, review hours and rework. The forecast and the actual go into a short log, with one line on what happened.
That one line should name which of Garg’s three areas, process, operator skill or context quality, was involved if the attempt went badly. A team should reach a capability-gap verdict for a class of work only after those three have been ruled out on more than one attempt. Otherwise a thin ticket gets recorded as a model limitation, which is the misreading Garg warns about.
The log is a record for the people deciding how to staff work. It is reviewed at planning time and kept off dashboards and performance reviews, which respects Garg’s warning about gaming. Its job is to make the next pricing conversation start from evidence: for this class of work, our forecasts and actuals look like this.
One team’s AI velocity result will not transfer to another team
The research also sets a boundary on how far any single result travels. In October 2024, Google researchers led by Elise Paradis published a randomized controlled trial with 96 full-time Google engineers working on one complex, enterprise-grade task. AI shortened their time on task by about 21%, with what the authors call a large confidence interval. They state that the effect size cannot be assumed to apply more broadly, or to carry across tools and over time.
So two careful trials produced results in opposite directions: about 21% faster at Google in mid-2024, 19% slower in mature open-source repositories in early 2025. METR’s authors examined 20 properties of their setting that could explain the slowdown, including the size and quality standards of the projects, and concluded the effect was unlikely to be mainly an artifact of their design. The setting matters more than any headline number. That supports Garg’s point that each team has to test its own work, and it adds a constraint: one task tells a team almost nothing. METR needed 246 tasks to produce its estimate. A team should collect forecast and actual pairs across a quarter for a given class of work before drawing a conclusion, and should expect the conclusion to expire when the tools change.
Why This Matters for Engineering Leadership
AI tooling is now a budget line that engineering leaders have to defend, and headcount requests compete with it. Garg’s honest position, that he cannot yet show higher spend buying proportional delivery, is where most leaders actually stand. A log of forecasts and actuals is what lets a leader move past that position with evidence on either side: keep funding the tools for the work where agents beat the forecast, and staff senior engineers without apology for the work where they did not. It also protects the engineers doing the trying. When an attempt with clean context and a fair review stalls, the log records evidence about the tool, and no one has to defend it as a personal failure.
FAQ
Why is AI spend a poor measure of AI coding velocity?
Spend shows that tools are being used. It bundles many behaviors into one number and cannot say whether delivery changed or why.
What did the METR randomized trial find?
In METR’s 2025 trial, 16 experienced open-source developers completed 246 tasks. With AI allowed they took 19% longer, while estimating afterwards that AI had cut their time by 20%.
Why write a forecast before an AI trial?
A forecast recorded before the work gives the result something fixed to compare against. Without it, teams judge the result by how the work felt, which the METR trial shows can point the wrong way.
Do AI coding tools speed developers up or slow them down?
Both results exist. Google’s 2024 trial with 96 engineers found about 21% less time on one enterprise task. METR found a 19% slowdown on mature open-source projects. The setting decides.
How many tasks does a team need before concluding anything?
More than one. A single task is an anecdote. A team should collect forecast and actual pairs across a quarter for one class of work before calling a capability gap.
Should the trial results go on a dashboard?
No. Kept as a reviewed log of forecasts and actuals, the record stays useful. Turned into a target, it invites gaming.
