Microsoft's AI doctor panel beat solo physicians 83% to 36%. Read the fine print first.

Microsoft says its AI diagnostic system is now more than twice as accurate as doctors on medicine's hardest cases. The paper behind that claim came out Thursday, October 22, in Science. The headline numbers are real. So is the fine print.

What it is. MAI-DxO (Microsoft AI Diagnostic Orchestrator) isn't a new model. It's a layer that runs on top of other companies' models and acts like a panel of virtual doctors: each agent proposes diagnoses, picks questions or tests, and challenges the others, over several rounds, while weighing what each test costs. Microsoft tested it with models from OpenAI, Google, Anthropic, Meta and DeepSeek; the best results came with OpenAI's GPT-5 and GPT-5.5.

The test. Microsoft built a benchmark, SDBench, that turns 304 New England Journal of Medicine case records and 325 emergency-department admissions (from the MIMIC-IV-ED dataset) into step-by-step cases. The diagnostician, human or AI, starts with a short summary and has to ask for history, order tests and decide when to commit. An AI "gatekeeper" answers the requests and an AI "judge" grades the final answer. Test costs come from a real hospital price list.

The numbers (Science paper (opens in a new tab)):

• Hard NEJM cases: MAI-DxO with GPT-5, 83.2%. Generalist physicians with search and reference tools, 36.1%; without them, 22.3%. Even the best single physician (51.8%) was beaten on both accuracy and cost.
• Emergency-department cases: 80.3% vs 46.8% for physicians.
• With a budget setting, it still hit 80.9% on NEJM cases at $2,241 in tests per case.
• Rerunning everything with GPT-5.5, without changing the prompts, kept MAI-DxO ahead.

The catch

• It's a simulation. Text only: no imaging, no waveforms, no pathology slides, no real patients. The paper itself says it measures reasoning "under idealized information conditions rather than bedside performance or patient outcomes."
• The doctors were generalists, working alone. Primary-care and hospital physicians, not the specialists or teams that usually handle cases this complex. The authors say they'd expect the gap to narrow against matched specialists. Physicians were tested on 56 held-out NEJM cases.
• The top score wasn't the cheap one. The 83.2% setting averaged $5,431 in tests per case; physicians with reference tools averaged $4,725. The cheaper settings are where the "lower cost" claim comes from.
• Much of the gain is the model. TechCrunch (opens in a new tab) notes GPT-5 alone with a basic prompt scored 78.9% on the hard cases; the orchestrator added about four points and cut costs. On emergency cases with GPT-5.5, both versions scored about 85%. The paper puts the orchestrator's average boost across models at 4.4 points (ED) and 7.9 points (NEJM).
• Who wrote it. Every author is a Microsoft employee holding company equity, and several are named inventors on a related Microsoft patent application. No outside funding.

What's next. Microsoft says it is open-sourcing SDBench for select NEJM cases (code on GitHub (opens in a new tab), MIT license) and will run a prospective study with Beth Israel Deaconess Medical Center in Boston, where consenting patients interact with the system alongside normal care. It's also working with Mayo Clinic on a healthcare foundation model planned for next year, CEO Mustafa Suleyman told TechCrunch.

Our take

This is a strong, peer-reviewed result about AI reasoning through a diagnosis step by step, and the cost-aware setup is a better test than multiple-choice licensing exams. It is not evidence that an AI beats your doctor. It beat individual generalists on curated text cases, graded by an AI judge, in a study run entirely by the company that sells the product. The hospital study is the one to watch.

What we did and didn't do: we read the Science paper, Microsoft AI's announcement and TechCrunch's interview coverage on October 23, 2026, and checked the SDBench repo exists. We didn't run SDBench or test MAI-DxO, which isn't publicly available. $MSFT. Not investment advice. Not medical advice. Not sponsored; no affiliate links.

MODEL Editors