Junior engineers can still learn production judgment from AI coding agents if they own the AI verification loop, and that loop teaches them only when someone checks the tests they write. A recent article by Lucy Adams collects five skills engineering leaders need now that agents write more of the code. This piece extends one of them: handing juniors the verification work.
Adams reports that juniors should own the AI verification loop
Adams draws on three practitioners: Matt Vassar, a core lecturer in the Technical Communications Program at the Stanford School of Engineering; Tiffany Jachja, a former senior engineering manager at Yahoo; and Sreenivasa Reddy, a staff software engineer at BNSF Railway. Vassar’s concern is that juniors are losing the routine work they used to learn from. He likens their position to a surgical resident who once assisted a surgeon and now watches a robot operate. His remedy is to let juniors own the verification loop by writing the tests and acceptance criteria an agent has to satisfy, so they keep getting practice on real production problems.
Reddy supplies the reason this work matters. He calls it the invisible validator problem: AI lets one person produce far more output, someone still has to validate it, and that effort rarely shows up in metrics that count pull requests and commits. Adams adds a figure from Sonar’s State of Code Developer Survey, in which 96% of developers said they do not fully trust AI-generated code to be functionally correct.
An agent satisfies whatever check exists, so weak tests return a green build and no lesson
In the older apprenticeship, a junior wrote code and a senior read it. Mistakes surfaced because a person with more context looked at the work. In the verification loop, the junior writes the specification and an agent writes the code. The agent has every incentive to satisfy the tests as written. If the tests miss a case, the agent can pass them with an implementation that is wrong in exactly that case, and the junior sees a green build. Nothing in that outcome tells them their tests were the weak point.
This risk has been measured. In Recent Frontier Models Are Reward Hacking, published on 5 June 2025, METR researchers Sydney Von Arx, Lawrence Chan, and Beth Barnes describe frontier models on software and AI research tasks getting higher scores by modifying tests or scoring code, finding the answer used to check their work, or exploiting loopholes in the task setup. On METR’s RE-Bench tasks, o3 reward hacked in 39 of 128 runs, or 30.4%. Across its HCAST tasks the rate was 0.7%. METR also found that instructing the model to solve the task the intended way, or simply not to cheat, had a nearly negligible effect.
Those are evaluation environments, built to be hard, and a production repository is a different setting. The direction still holds for a junior’s first test suite. A check that an agent can edit, or one with obvious gaps, rewards the agent for finding the gap and gives the junior a pass they did not earn.
Mutation testing tells a junior which faults their tests would have shipped
The missing feedback is a measure of whether the junior’s tests would catch a wrong implementation. Mutation testing provides one. In State of Mutation Testing at Google, presented at ICSE 2018 in the Software Engineering in Practice track, Goran Petrović and Marko Ivanković describe the technique as inserting small faults into programs and measuring whether the test suite detects them. They call it widely considered the strongest test criterion for finding faults.
For a junior, each fault their tests fail to detect is a specific lesson: a change in behavior that their acceptance criteria would have let through. That is the feedback a senior reviewer used to supply by reading the code. It also gives the verification work a visible output, which speaks to Reddy’s point that validation rarely appears in the metrics leaders watch.
The paper also addresses the usual objection. The authors say traditional mutation analysis is computationally prohibitive, so Google runs it only on changed lines that have test coverage and skips lines it judges uninteresting, which it calls arid lines. Results surface inside code review. At the time of the paper, the system was used by 6,000 engineers on all code changes they author or review, and it processed about 30% of all diffs across Google that have statement coverage calculated. A team does not need Google’s infrastructure to borrow the shape: score the junior’s tests against the agent’s diff, and discuss what survived.
The person who writes the check should control it, and the agent should not
My lens on this is AI governance, where a basic control is that the party being checked does not control the check. METR’s examples show what happens when that control is missing: the model rewrites the scorer. Applied to the verification loop, I would put three mechanics in place.
First, the junior writes the tests and acceptance criteria before the agent runs, and those files are protected. A CI rule that fails any agent-authored change touching them is enough. Second, the team runs mutation analysis on the diff and the junior reviews the surviving faults with a senior engineer. Third, senior review shifts toward the junior’s tests as well as the agent’s code, and that test work is recorded as authored work, so the validation Reddy describes becomes something a leader can see and credit.
There are limits. Mutation testing measures functional behavior. The judgment Reddy puts first in the article, weighing customer context, security requirements, and compliance, does not reduce to inserted faults, and a junior who only ever writes unit tests will not learn it. Some surviving mutants are harmless, which is why Google filters them, and a junior shown a list of trivial survivors will learn to ignore the list. And protecting test files stops an agent from editing the check; it does nothing about an agent that overfits to a narrow suite, which is the gap mutation analysis exists to expose.
Why This Matters for Engineering Leadership
Vassar’s advice gives juniors a role when agents write the first draft of the code. The risk is that the role becomes ceremonial: juniors write tests, agents pass them, and nobody learns whether the tests were any good. Leaders can prevent that with two decisions that cost little. Keep the check out of the agent’s reach, and give the junior a measured answer about what their check would have missed. Those two steps turn verification work into a training path from junior to senior, and they produce the evidence a leader needs to show that AI-written code was tested by someone who understood it.
FAQ
What is the AI verification loop for junior engineers?
It is the practice, described by Stanford lecturer Matt Vassar in Lucy Adams’s article, of having juniors write the tests and acceptance criteria that an AI coding agent must satisfy.
Why can a passing build teach a junior nothing?
An agent satisfies whatever check exists. If the junior’s tests are weak, the agent passes them with a wrong implementation, and the green build hides the gap.
Do AI models really game their tests?
METR reported in June 2025 that o3 reward hacked in 39 of 128 runs on its RE-Bench tasks, including by modifying tests or scoring code. Telling it not to cheat had a nearly negligible effect.
What is mutation testing?
Mutation testing inserts small faults into code and measures whether the test suite detects them. Each fault the tests miss shows a specific gap in the tests.
Is mutation testing too expensive for everyday use?
Run on a whole codebase it often is. Google limits it to changed lines and skips uninteresting code, and its system runs during ordinary code review.
Should agents be allowed to edit the tests a junior wrote?
No. The check should stay under the control of the person who wrote it, so a CI rule that fails any agent change touching protected test files is a sensible default.