When a conventional background job fails, the application usually has a relatively small collection of predefined responses. It can retry the operation, move the job to a failure queue, or notify an operator. Although implementing reliable recovery is difficult, the application generally knows what the job was supposed to do.
Autonomous agents introduce another layer of uncertainty. An agent might fail because a tool becomes unavailable, because it has misunderstood the task, or because the investigation has reached a point where its next decision is ambiguous. In some cases, the operation may have succeeded even though the agent never received confirmation.
This becomes important in a system such as Third Moment Research's, where thousands of agents conduct long-running computational investigations. Each agent can generate additional work, use multiple tools, and make decisions based on the results of previous experiments.
According to OpenAI's case study, Third Moment addresses incomplete work by running Codex inside another harness powered by GPT-6 Sol or GPT-6.1 Sol. When an investigation stops prematurely, the supervisor examines its execution trajectory and determines whether to retry the work, terminate it, or involve a human.
That design separates execution from recovery decisions. It does not, however, eliminate the underlying distributed-systems problems.
The difference between task failure and infrastructure failure
Consider a research agent investigating the performance of a trading strategy. It submits a simulation, receives a job identifier, and waits for the results.
Ten minutes later, the request to retrieve the result times out.
Several explanations are possible. The simulation service may have crashed. The simulation may still be running. The simulation may have completed successfully, but the response was lost. Alternatively, the agent may have supplied an invalid configuration that the simulation could not process.
These situations require different responses, yet a simple timeout exception does not distinguish between them.
If the agent immediately submits the same simulation again, it may duplicate expensive work that has already completed. If it abandons the investigation, it may discard a valid result. If it repeatedly retries an invalid configuration, it may waste hours without making progress.
The execution system therefore needs a richer representation of failure than a Boolean success-or-failure flag.
| Failure class | Example | Suitable recovery behavior |
|---|---|---|
| Transient infrastructure failure | Network request temporarily unavailable | Bounded retry with backoff |
| Uncertain completion | Worker stopped after submitting a job | Inspect job state before retrying |
| Invalid task output | Experiment failed validation | Revise the experiment or hypothesis |
| Resource exhaustion | Compute or token budget exceeded | Pause, reduce scope, or terminate |
| Policy violation | Unauthorized network or filesystem access | Block and record the operation; escalate when required |
| Repeated non-progress | Agent keeps revisiting the same unsuccessful approach | Terminate or require a revised plan |
The correct response may depend on the task's history. A failure on the first attempt is different from the same failure occurring after fifteen retries.
It also depends on the consequences of repeating an operation.
Why idempotency matters for agent tools
An operation is idempotent when repeating it has the same intended effect as executing it once.
For example, retrieving the current status of an experiment is normally safe to repeat. Creating a new experiment, allocating compute resources, modifying a database, or sending an external notification may not be.
Suppose an agent submits a trading simulation through an API. The request reaches the server, which creates the simulation successfully. Before the response reaches the agent, the network connection fails.
From the agent's perspective, the operation appears unsuccessful. From the server's perspective, the operation has already happened.
A naive retry creates a second simulation.
A safer API design allows the caller to supply a stable idempotency key associated with the logical operation. If the agent retries with the same key, the service can return the original job rather than creating another one.
Amazon's engineering guidance on idempotent APIs discusses this problem in the context of distributed services. The same principle applies when an AI agent becomes the caller.
The important requirement is that retry semantics are enforced by the receiving service, not merely suggested in the model's prompt.
Building a recovery state machine
A long-running agent should have an explicit lifecycle independent of its conversational history.
An investigation may be pending, running, waiting for an external operation, undergoing recovery, awaiting human review, completed, or terminated.
These states should be durable. If an execution process disappears, the scheduler must be able to determine whether the task is eligible for resumption and which operations remain unresolved.
Figure 3: Recommended failure-recovery state machine
flowchart TD
A["Task Running"] --> B{"Execution Outcome"}
B -->|"Success"| C["Validate Result"]
B -->|"Timeout / Worker Failure"| D["Inspect Durable Job State"]
B -->|"Invalid Result"| E["Revise Task Plan"]
B -->|"Policy Violation"| F["Block and Record"]
D --> G{"Operation Completed?"}
G -->|"Yes"| C
G -->|"No"| H{"Retry Allowed?"}
G -->|"Unknown"| I["Escalate / Reconcile State"]
H -->|"Yes"| A
H -->|"No"| I
E --> J{"Revision Budget Available?"}
J -->|"Yes"| A
J -->|"No"| I
C --> K["Mark Complete"]
F --> I
Illustrative recovery policy, not Third Moment's published implementation.
A model-based supervisor fits naturally into this design. It can evaluate whether an invalid result calls for a different hypothesis, whether an investigation has reached diminishing returns, or whether additional experiments would be informative.
But certain decisions should remain outside its discretion. A supervisor should not be able to ignore security restrictions, increase its own resource allocation without authorization, or retry an operation that the policy system has explicitly prohibited.
Execution traces must capture causality
Third Moment used thousands of recorded execution trajectories to identify inefficiencies in its research environment. The same records are valuable for diagnosing failures.
An agent trace should make it possible to reconstruct which objective produced a particular action, which agent initiated it, which tools were invoked, and what the environment returned.
For complex workflows, individual tool-call logs are insufficient. The system also needs the causal relationships between operations.
Suppose a subagent modifies a simulation parameter, another subagent launches a backtest using that parameter, and the parent agent later rejects the result. A useful trace should connect the rejected experiment to the configuration change that produced it.
OpenTelemetry's evolving generative AI semantic conventions provide a starting point for representing agent invocations, workflows, planning, and tool execution as related spans. These conventions are still evolving, and domain-specific execution artifacts generally require additional instrumentation.
For operational debugging, I would record at least the following information:
- A stable investigation ID and parent-child relationships between agents.
- Model and tool versions used for each operation.
- Resource consumption and execution duration.
- Input and output artifact references.
- Failure classification, retry count, and termination reason.
- Supervisor decisions and any human approvals.
An important caution is that traces can contain sensitive tool arguments, credentials, customer data, or proprietary research. Observability systems therefore require their own access controls, retention policies, and redaction mechanisms.
Measuring whether recovery actually works
A system that eventually completes every task after unlimited retries is not necessarily reliable.
It might be consuming excessive compute, silently discarding failed experiments, or creating new tasks faster than it resolves old ones.
Reliability measurement should distinguish successful completion from the cost and correctness of reaching completion.
Useful metrics include task completion rate, median recovery time, retry amplification, percentage of investigations requiring human intervention, and the proportion of results passing independent validation.
At large scale, I would also examine whether failures are correlated. Ten agents failing independently is different from ten thousand agents encountering the same broken service.
A supervisor capable of interpreting an individual agent's trajectory will not solve a shared infrastructure outage by repeatedly restarting every affected investigation.
Why this matters
Third Moment's disclosed architecture demonstrates an important approach to handling incomplete autonomous work: introduce a supervisory layer capable of inspecting execution history and making recovery decisions.
The broader production requirement is more demanding. A reliable agent platform needs persistent execution state, explicit failure categories, safe retry semantics, independently enforced policies, and visibility into whether recovery decisions actually improved the outcome.
An agent should not continue simply because it can. It should continue because the infrastructure can establish that another attempt is appropriate, safe, and worth the resources it will consume.
Sources: OpenAI — Third Moment Research · AWS — Making Retries Safe with Idempotent APIs · OpenTelemetry Semantic Conventions
Related: Multi-Agent Orchestration Architecture | Long-Running Agent State Management