Microsoft Says Its AI Beat Doctors on the Hardest Cases in Medicine
Microsoft says its MAI-DxO system diagnosed 83% of the toughest NEJM cases in a Science study, against 36% for generalist doctors. The catch is a text-only test, no specialists, and no approval for clinical use.
On the nastiest diagnostic puzzles medicine publishes, the doctors in Microsoft's study got the answer right about a third of the time. Microsoft's AI got it right 83% of the time. That is the headline number from a paper the company's researchers published Thursday in Science, and Microsoft AI chief Mustafa Suleyman is already calling it a step toward "medical superintelligence."
It is a striking result. It is also a test Microsoft designed and ran itself, against a carefully chosen kind of doctor, and patients deserve to hear that half as loudly.
The system is called the Microsoft AI Diagnostic Orchestrator, or MAI-DxO. Instead of asking one model for an instant answer, it operates, in the words of Microsoft's announcement, like "a panel of virtual physicians" that propose diagnoses, pick questions and tests, and challenge each other over several rounds. In the main results it ran on OpenAI's GPT-5.
Microsoft tested it on its own benchmark, SDBench, built from 304 New England Journal of Medicine case records and 325 emergency admissions from the MIMIC-IV-ED database. Each case starts with a short vignette, and the diagnostician, human or machine, must ask questions and order tests one step at a time.
According to the paper, MAI-DxO hit 83.2% accuracy on the NEJM cases. Generalist physicians scored 36.1% with search and reference tools and 22.3% without them. On the more ordinary emergency cases, the AI scored 80.3% against 46.8% for physicians. Even the best individual doctor on the NEJM set, at 51.8%, was beaten on both accuracy and cost, the researchers wrote.
Cost is where Microsoft's marketing runs ahead of its own data. Unconstrained, MAI-DxO actually spent more on tests than the doctors, averaging $5,431 per NEJM case against $4,725 for physicians with references. Only when the researchers capped its budget did it get cheap, reaching 80.9% at $2,241 and 79.6% at $1,303.
The fine print patients should read
The authors spell out the limits themselves. Every case was text only, with no imaging, waveforms or pathology slides. The humans were individual generalists, primary care, hospital and emergency doctors, not the specialists or multidisciplinary teams who normally handle NEJM teaching cases. The researchers say they would expect the gap to narrow against matched specialists. The benchmark also measured diagnosis alone, not treatment, safety or patient outcomes.
"These comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice." Microsoft researchers, writing in Science
There is also the question of who graded the homework. Every author is a Microsoft employee holding company equity, eight are named inventors on a related patent application, and the study received no outside funding, according to the paper's disclosures. TechCrunch noted that much of the improvement appears to come from stronger underlying models, with the orchestration layer adding gains on top.
When this work first appeared as a 2025 preprint, MIT's David Sontag told Wired the findings deserved some caution because doctors could not use extra tools. The Science version adds a group with reference tools, which answers part of that critique.
Nobody should mistake any of this for a product. When it released the preprint, Microsoft said the work was "not yet approved for clinical use" and would need "rigorous safety testing, clinical validation, and regulatory reviews." Suleyman struck the same note this week.
"It needs clinical validation. That is a leap, but this is a major milestone along that journey." Mustafa Suleyman, CEO of Microsoft AI, to The Verge
That leap starts with Beth Israel Deaconess Medical Center in Boston, where Microsoft says a prospective study will let consenting patients interact with the system while their care continues as normal, with its calls compared against doctors' independent assessments. Dominic King, Microsoft AI's health lead, told The Verge it will likely "get underway in the next couple of months."
Microsoft is not alone here. Google published a Lancet study of its AMIE diagnostic AI in urgent care this month, and researchers are pitching tools that read survival odds from a selfie. SDBench is a smarter test than multiple choice exams. The real exam is a crowded emergency room on a Friday night, and Microsoft has not sat it yet.
