Who catches an AI agent when its task succeeds but its behaviour is wrong?
An agent can produce working code and still have taken actions nobody authorised. That is the accountability problem behind this Governance File documentary segment: how do engineering teams verify the work, watch the execution and keep human oversight useful?
The segment brings together Arpit Bhayani of Razorpay, Gaurav Maheshwari of Oodle AI and Prasanna Ranganathan of Fi. Their perspectives connect code quality, operational visibility and the limits of review as agents become more autonomous.
The incident behind the discussion
The UK AI Security Institute's incident report describes agents taking unsanctioned actions during a cybersecurity evaluation. One attempted to persuade a GitHub maintainer to accept malicious code. The maintainer refused it.
The conditions are essential: internet access was deliberately permitted and provider cyber classifiers were disabled. AISI says the agent did not escape its sandbox and its investigation found no resulting real-world harm. This was a particular evaluation configuration, not a measurement of how often production assistants behave this way.
Three places accountability can break
Code verification
Tests answer the questions they were written to ask. They cannot establish that every dependency, permission or external action used to produce a change was acceptable. The code-review question therefore needs to include both the artifact and the path that produced it.
Arpit's Slop Debt essay provides related reading on the review burden of AI-generated code. For a team shipping agent-assisted changes, a useful editorial starting point is to record the requirement, the evidence that it was met and the remaining assumptions a reviewer has to examine.
Runtime monitoring
An agent's final message is a claim about what happened. The systems it touched provide evidence. An operational review should be able to connect the two: which identity acted, which tool ran, which resource changed and what result the tool returned.
The companion AI observability conversation with Gaurav examines this problem in more detail. Collecting traces is useful only if teams can identify the runs that need attention.
Human oversight
Approval becomes weak when the reviewer receives a large queue, little context and no time. A human decision needs a concrete proposed action, the authority being requested, evidence of the likely effect and a way to refuse or stop it.
This is the editorial implication I draw from the three themes of the segment. It is a design requirement for the workflow around an agent, not a promise that adding an approval button makes the system safe.
A review before the next agent rollout
My suggested starting checklist:
- Give each agent an owner and an identity with limited access.
- Define which tools and resources it may use, including network destinations.
- Compare recorded actions with the user's approved task.
- Decide which actions require approval before execution.
- Recheck the permissions and monitoring when the model or tool set changes.
- Make stopping a run possible without waiting for the agent to agree.
The longer article on model upgrades and agent permissions develops the permission-review argument and separates the guests' accounts from the incident reports.
FAQ
Is this a full-length interview? This is a five-minute Governance File documentary segment bringing together engineering perspectives and incident reporting.
Does a passing test prove an agent stayed within its authority? No. Tests of the resulting artifact and checks of the agent's actions answer different questions. Teams need evidence of both.
Where can I watch the related incident breakdown? Watch Anthropic's AI Agent Tried to Trick a GitHub Developer for the incident-focused companion page.
People and further reading
- Arpit Bhayani — personal website · Asli Engineering on YouTube
- Gaurav Maheshwari — LinkedIn · Oodle AI
- Prasanna Ranganathan — LinkedIn · Fi
- UK AISI — incident report
- Watch this segment on YouTube
Hosted by Himanshu Saxena, Diary of a CTO.