One of the most ambitious elements of Third Moment Research's work is its attempt to build agents that become better at their assigned tasks through repeated use.
According to OpenAI's case study, the company analyzes recorded agent behavior to identify recurring weaknesses. It then generates synthetic scenarios designed around those problems and uses the scenarios to improve the agents.
The process is intended to repeat as new problems emerge.
This is different from an agent simply reflecting on a previous response and deciding to try again. Third Moment is building an improvement process around recorded execution trajectories, controlled scenarios, and repeated evaluation.
The approach is potentially important because specialized organizational work often has clear definitions of success that general-purpose AI benchmarks do not capture.
A trading strategy can be evaluated against historical data. A software optimization can be measured against a performance baseline. A data-processing workflow can be tested for correctness, latency, and resource consumption.
These environments provide feedback that can guide agent improvement, although producing trustworthy feedback is itself a difficult engineering problem.
What does it mean for an agent to improve?
An autonomous agent is not just a language model. Its performance depends on the combination of the model, the instructions it receives, its tools, the information available in its environment, and the software controlling execution.
An observed improvement could therefore originate in several places.
The underlying model might become better at reasoning about unfamiliar problems. A modified tool could execute operations more efficiently. A change to the orchestration layer might reduce unnecessary retries. An improved evaluation procedure might help the system reject unreliable results earlier.
These changes can all make the overall agent system more effective without necessarily altering the model's weights.
Third Moment's backtesting optimization provides a concrete example. By improving caching and memory packing, Codex reduced the runtime of a computational bottleneck from hours to minutes.
That change potentially increases the number of experiments agents can conduct, even if their underlying reasoning capabilities remain unchanged.
The public case study does not disclose the full mechanism used for every type of agent improvement, so it would be incorrect to assume that the entire process consists of autonomous model retraining.
The role of execution trajectories
The starting point for Third Moment's feedback loop is recorded agent behavior.
An execution trajectory captures the sequence of actions taken during an investigation and the responses returned by the environment.
For example, consider an agent trying to optimize a software service. Its trajectory might show that it repeatedly modifies the same function, runs an expensive benchmark, receives an inconclusive result, and returns to the previous implementation.
If similar patterns occur across many investigations, they may reveal a systematic weakness in the agent's approach.
Perhaps the agent does not select sufficiently informative experiments. Perhaps it lacks access to relevant profiling data. Perhaps its instructions encourage local changes without examining broader bottlenecks.
The value of analyzing many trajectories is that recurring patterns become easier to distinguish from isolated mistakes.
But identifying a recurring failure is only the beginning.
The system still needs to determine why the failure occurs and whether a proposed change actually resolves the underlying problem.
Using synthetic scenarios to test weaknesses
Third Moment describes generating thousands of synthetic scenarios around recurring agent weaknesses.
A synthetic scenario is a constructed task or environment intended to reproduce a particular type of challenge.
Suppose agents repeatedly struggle to distinguish meaningful performance improvements from ordinary measurement noise. A synthetic evaluation could present several benchmark results, including cases where an apparent improvement is statistically insignificant.
A successful agent would need to recognize the uncertainty rather than automatically selecting the configuration with the lowest observed latency.
Alternatively, if agents frequently mishandle interrupted tool operations, synthetic scenarios could reproduce partial failures, delayed responses, and ambiguous completion states.
These scenarios can be used to evaluate changes to the agent system.
Figure 7: An agent improvement and evaluation cycle
flowchart TD
A["Real Agent Executions"] --> B["Collect Execution Trajectories"]
B --> C["Identify Recurring Failure Patterns"]
C --> D["Generate Synthetic Scenarios"]
D --> E["Development Evaluation"]
E --> F["Modify Agent System"]
F --> G["Independent Validation"]
G --> H{"Improvement Generalizes?"}
H -->|"Yes"| I["Controlled Deployment"]
H -->|"No"| C
I --> A
Conceptual improvement process. The independent validation stage is an engineering recommendation; the case study does not disclose Third Moment's complete evaluation methodology.
The critical stage is independent validation.
If the same scenarios are repeatedly used to develop and evaluate an agent, the system can become increasingly effective at passing those particular tests without improving on unfamiliar problems.
Why thousands of experiments can produce misleading results
Third Moment's initial focus on quantitative finance provides an unusually clear example of the difficulty.
Suppose an agent is tasked with improving a trading strategy. It generates hundreds of candidate signals and evaluates each against historical market data.
If one strategy produces higher returns than the baseline, the agent might conclude that it has discovered a useful improvement.
But repeated experimentation creates a statistical selection problem.
Even when candidate strategies have no genuine predictive power, some will appear successful by chance.
Consider a simplified example involving 100 independent statistical tests. If every null hypothesis is true and each test uses a 5% significance threshold, the expected number of false positives is five.
The probability of observing at least one false positive is:
P(at least one false positive) = 1 - (1 - 0.05)^100 ≈ 99.4%
Under the independence assumption, this is approximately 99.4%.
Real trading experiments are often correlated, so this calculation is illustrative rather than a direct estimate of the false-discovery rate in quantitative research. Nevertheless, it shows why selecting the best-performing result from a large collection of experiments is not sufficient evidence of genuine improvement.
Figure 8: Expected false positives under repeated testing
| Number of independent tests | Expected false positives at 5% |
|---|---|
| 10 | 0.5 |
| 50 | 2.5 |
| 100 | 5 |
| 500 | 25 |
| 1,000 | 50 |
Illustrative statistical expectation assuming all null hypotheses are true.
This becomes particularly important when the system uses previous results to design new experiments.
Once the agent repeatedly examines a dataset and adapts its hypotheses based on the results, the dataset is no longer an independent source of evaluation.
The research process can gradually overfit the historical environment.
What a reliable evaluation framework requires
For computational research, evaluation needs to determine whether an apparent improvement survives outside the conditions used to discover it.
In quantitative finance, that may require temporally separated evaluation datasets, walk-forward testing, realistic transaction-cost assumptions, and careful treatment of repeated hypothesis testing.
For software engineering, the requirements differ. A code optimization should preserve functional correctness while improving performance. Benchmark conditions must remain comparable, and the system should avoid accepting changes that perform well only on a narrow collection of inputs.
Across domains, several principles remain consistent:
- Preserve experimental history. Record unsuccessful experiments as well as successful ones, so the evaluation process accounts for the search that produced a result.
- Separate development from validation. Do not use the same collection of cases indefinitely for both improving and judging the system.
- Measure generalization. Evaluate performance on problems or environments that were not used to guide the changes.
- Track regressions. An improvement on one task category should not automatically justify reduced performance elsewhere.
- Require reproducibility. Important results should be supported by recorded configurations, artifacts, and independent reruns where appropriate.
These requirements may become more important as agents grow capable of conducting larger numbers of experiments.
A system that can generate and test thousands of hypotheses is useful only if it can reliably distinguish meaningful findings from noise.
Why collaboration adds another level of difficulty
Third Moment's longer-term ambition is to operate hundreds of thousands of collaborating subagents.
Collaboration could make research more efficient by allowing different agents to investigate independent hypotheses simultaneously. It could also allow specialized agents to evaluate one another's work.
But collaboration introduces dependencies between experiments.
Suppose one agent identifies a promising trading signal and another begins optimizing a strategy that incorporates it. A third agent later discovers that the original signal was caused by a data-quality problem.
The second agent's work may now be invalid.
A collaborative research platform needs to track these relationships so that changes to upstream assumptions can invalidate or trigger re-evaluation of dependent results.
This resembles dependency management in build systems and distributed workflow engines, except that the relationships may be created dynamically during research.
The system also needs policies for resolving contradictory findings. Two agents may evaluate the same hypothesis using different data or methods and reach different conclusions.
The appropriate response is not necessarily to let a supervisory model choose whichever explanation appears more convincing. The system may need to launch additional controlled experiments or refer the disagreement for human review.
The distinction between learning and producing more work
Third Moment's reported progress suggests that newer models can conduct more sophisticated computational investigations, including experiments that human researchers did not explicitly prescribe.
The company describes agents identifying research directions its own team had not initially considered.
That is an important capability, but the long-term measure of success will be whether the resulting research process consistently produces independently validated improvements.
More experiments do not necessarily mean better discoveries. Longer sessions do not necessarily mean greater reliability. And successful synthetic evaluations do not automatically establish that a system has become more capable on unfamiliar tasks.
The challenge for self-improving agents is to build a feedback process in which observed failures lead to changes that survive independent scrutiny.
If that can be achieved reliably, the value of autonomous agents would extend beyond executing predefined work. They could become useful participants in the process of discovering how the work itself should be improved.
Sources: OpenAI — Third Moment Research · OpenTelemetry — GenAI Semantic Conventions
Related: Third Moment's 10,000-Agent System | Multi-Agent Orchestration Architecture