Microsoft’s MAI-DxO diagnostic system correctly diagnosed 80.3% of emergency department admissions and 83.2% of complex New England Journal of Medicine cases in a peer-reviewed study published in Science on October 22, 2026. Generalist physicians given the same cases scored 46.8% and 36.1%. The paper, “Sequential diagnosis with language models,” appears in volume 394, issue 6822, with 15 authors. Its authors describe the results as obtained in a controlled benchmark setting, and the system has not yet been tested on live patients.
MAI-DxO Works a Case Step by Step, the Way a Clinician Does
MAI-DxO, the Microsoft AI Diagnostic Orchestrator, runs a language model as a simulated panel of physicians. According to Microsoft AI’s announcement of the paper, each virtual panel member proposes diagnoses, picks useful questions or tests, and challenges the emerging view, and the orchestrator runs several rounds of that debate while weighing each test’s likely value against its cost. The system starts with a brief clinical presentation and has to ask for everything else. Microsoft says the gains held, and improved, when it repeated the evaluation with newer models, including GPT-5.5, without changing the orchestration method.
SDBench Charges the AI for Every Test It Orders
The Sequential Diagnosis Benchmark, or SDBench, is the part of the study that transfers beyond medicine. It turns 304 NEJM case records and a set of typical emergency presentations from the MIMIC-IV-ED database into interactive cases. A component Microsoft calls the Gatekeeper releases information only when the model or physician requests it, a Judge scores the final diagnosis, and a cost estimator prices each test. Microsoft says the benchmark measures diagnostic accuracy, the evidence requested and the estimated test cost, so it assesses the path taken as well as the answer. Microsoft contrasts this with licensing-exam style benchmarks, where all information arrives up front and a single answer is picked.
The Headline Accuracy and the Cost Savings Come From Different Settings
Microsoft’s announcement says MAI-DxO is more accurate than physicians at half or even a third of the cost. Read closely, those claims describe a range of configurations. On NEJM cases, Microsoft says one configuration tuned more heavily for cost reached 81% accuracy at less than half the cost of comparable systems, below the 83.2% headline figure. Microsoft also says no other human or AI approach in the study was both more accurate and less costly than MAI-DxO across both datasets. The physician comparison has limits worth stating. The doctors were generalists, and on the NEJM cases they scored 22% without reference resources and 36.1% with textbooks and the internet. The cases came from documented records with known diagnoses.
Real Patients Come Next, at Beth Israel Deaconess
Microsoft says a new partnership with Beth Israel Deaconess Medical Center in Boston will run a prospective study in which patients interact with the system directly, with its performance measured against physicians’ independent assessments. It also says it is open-sourcing SDBench for selected NEJM cases, including the benchmark, the Gatekeeper, the Judge and the cost-estimation code, so others can reproduce the evaluation and test other models.
Why This Matters for Engineering Leadership
SDBench is a template for evaluating any AI agent your team builds or buys. An evaluation that hands the model a complete context and grades only the final answer measures a task the agent will never face in production, where it has to decide which tool to call, which record to fetch and when to stop. Build your evaluation harness the way Microsoft built this one. Give the agent a thin starting state and make it request everything else through a gatekeeper that logs each request. Put a price on every tool call, whether that is tokens, API fees, human review time or latency. Then report accuracy and cost side by side for each configuration you test, and plot them as a frontier. The MAI-DxO paper shows why the last step matters: the most accurate setting and the cheapest good-enough setting were different runs, and a vendor quoting both numbers in one sentence is quoting two products. When a supplier shows you an agent benchmark, ask which configuration produced each figure, what each action cost, and whether the evaluation released information step by step or all at once.
FAQ
What is MAI-DxO?
MAI-DxO, the Microsoft AI Diagnostic Orchestrator, is a system that runs a language model as a simulated panel of physicians. It proposes diagnoses, chooses questions and tests, and revises its view as results arrive.
How accurate was MAI-DxO in the Science study?
It reached 83.2% on 304 New England Journal of Medicine case records and 80.3% on emergency department admissions from MIMIC-IV-ED. Generalist physicians scored 36.1% and 46.8% on the same sets.
What is SDBench?
The Sequential Diagnosis Benchmark starts each case with a short presentation and releases further information only when asked. It scores the final diagnosis, the evidence requested and the estimated cost of the tests ordered.
Is MAI-DxO cheaper than doctors?
Microsoft says cost-optimised configurations reach the right diagnosis with fewer tests, including one at 81% accuracy on NEJM cases for less than half the cost of comparable systems. Accuracy and cost depend on which configuration is run.
Has MAI-DxO been tested on real patients?
Not yet. The paper describes a controlled benchmark built from documented cases, and Microsoft says a prospective study with Beth Israel Deaconess Medical Center in Boston will have patients use the system directly.
What should engineering leaders take from the study?
Evaluate AI agents the way SDBench evaluates diagnosis. Withhold information until the agent asks, attach a cost to every tool call, and report accuracy and cost together for each configuration.
