# Sequential Diagnosis with Language Models

arXiv: https://arxiv.org/abs/2506.22405 · HTML: https://mlsanity.com/paper/2506.22405/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 13 of 13)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks](https://mlsanity.com/paper/2505.23802/) (2025)
- [Superhuman performance of a large language model on the reasoning tasks of a physician](https://mlsanity.com/paper/2412.10849/) (2024)
- [From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond](https://mlsanity.com/paper/2411.03590/) (2024)
- [The Optimization Paradox in Clinical AI Multi-Agent Systems](https://mlsanity.com/paper/2506.06574/) (2025)
- [Time-Critical Reasoning: Representations and Application](https://mlsanity.com/paper/1302.1548/) (2013)

## Built on by (most cited first, 30 of 60)

- [MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Framework](https://mlsanity.com/paper/2508.14880/) (2025)
- [Byzantine-Robust Decentralized Coordination of LLM Agents](https://mlsanity.com/paper/2507.14928/) (2025)
- [The Illusion of Readiness in Health AI](https://mlsanity.com/paper/2509.18234/) (2025)
- [The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models](https://mlsanity.com/paper/2509.10970/) (2025)
- [Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications](https://mlsanity.com/paper/2508.00669/) (2025)
- [The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability](https://mlsanity.com/paper/2509.24958/) (2025)
- [Evolving Interactive Diagnostic Agents in a Virtual Clinical Environment](https://mlsanity.com/paper/2510.24654/) (2025)
- [EvoClinician: A Self-Evolving Agent for Multi-Turn Medical Diagnosis via Test-Time Evolutionary Learning](https://mlsanity.com/paper/2601.22964/) (2026)
- [HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents](https://mlsanity.com/paper/2606.31179/) (2026)
- [MedKGI: Iterative Differential Diagnosis with Medical Knowledge Graphs and Information-Guided Inquiring](https://mlsanity.com/paper/2512.24181/) (2025)
- [OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries](https://mlsanity.com/paper/2509.02594/) (2025)
- [Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges](https://mlsanity.com/paper/2510.09404/) (2025)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs](https://mlsanity.com/paper/2605.06915/) (2026)
- [Leveraging Imperfection with MEDLEY A Multi-Model Approach Harnessing Bias in Medical AI](https://mlsanity.com/paper/2508.21648/) (2025)
- [ABRA: Agent Benchmark for Radiology Applications](https://mlsanity.com/paper/2605.11224/) (2026)
- [Sakana Fugu Technical Report](https://mlsanity.com/paper/2606.21228/) (2026)
- [VCBench: Benchmarking LLMs in Venture Capital](https://mlsanity.com/paper/2509.14448/) (2025)
- [Generalist Foundation Models Are Not Clinical Enough for Hospital Operations](https://mlsanity.com/paper/2511.13703/) (2025)
- [Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters](https://mlsanity.com/paper/2604.24710/) (2026)
- [CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics](https://mlsanity.com/paper/2605.09584/) (2026)
- [ResidencyRL: Reinforcement Learning in Simulated Clinical Environments](https://mlsanity.com/paper/2608.07418/) (2026)
- [On the Robustness of Answer Formats in Medical Reasoning Models](https://mlsanity.com/paper/2509.20866/) (2025)
- [Batch-of-Thought: Cross-Instance Learning for Enhanced LLM Reasoning](https://mlsanity.com/paper/2601.02950/) (2026)
- [Emulating Clinician Cognition via Self-Evolving Deep Clinical Research](https://mlsanity.com/paper/2603.10677/) (2026)
- [EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs](https://mlsanity.com/paper/2605.30637/) (2026)
- [MedCoAct: Confidence-Aware Multi-Agent Collaboration for Complete Clinical Decision](https://mlsanity.com/paper/2510.10461/) (2025)
- [Ask WhAI:Probing Belief Formation in Role-Primed LLM Agents](https://mlsanity.com/paper/2511.14780/) (2025)
- [How people use Copilot for Health](https://mlsanity.com/paper/2604.15331/) (2026)
- [DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs](https://mlsanity.com/paper/2605.23629/) (2026)

## Abstract (arXiv)

Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes and multiple-choice questions that fail to reflect the complexity and nuance of evidence-based medicine in real-world settings. In clinical practice, physicians iteratively formulate and revise diagnostic hypotheses, adapting each subsequent question and test to what they've just learned, and weigh the evolving evidence before committing to a final diagnosis. To emulate this iterative process, we introduce the Sequential Diagnosis Benchmark, which transforms 304 diagnostically challenging New England Journal of Medicine clinicopathological conference (NEJM-CPC) cases into stepwise diagnostic encounters. A physician or AI begins with a short case abstract and must iteratively request additional details from a gatekeeper model that reveals findings only when explicitly queried. Performance is assessed not just by diagnostic accuracy but also by the cost of physician visits and tests performed. We also present the MAI Diagnostic Orchestrator (MAI-DxO), a model-agnostic orchestrator that simulates a panel of physicians, proposes likely differential diagnoses and strategically selects high-value, cost-effective tests. When paired with OpenAI's o3 model, MAI-DxO achieves 80% diagnostic accuracy--four times higher than the 20% average of generalist physicians. MAI-DxO also reduces diagnostic costs by 20% compared to physicians, and 70% compared to off-the-shelf o3. When configured for maximum accuracy, MAI-DxO achieves 85.5% accuracy. These performance gains with MAI-DxO generalize across models from the OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama families. We highlight how AI systems, when guided to think iteratively and act judiciously, can advance diagnostic precision and cost-effectiveness in clinical care.