# Superhuman performance of a large language model on the reasoning tasks of a physician

arXiv: https://arxiv.org/abs/2412.10849 · HTML: https://mlsanity.com/paper/2412.10849/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 10 of 10)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?](https://mlsanity.com/paper/2409.15277/) (2024)
- [Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine](https://mlsanity.com/paper/2401.08396/) (2024)
- [From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond](https://mlsanity.com/paper/2411.03590/) (2024)
- [Multimodal Foundation Models Exploit Text to Make Medical Image Predictions](https://mlsanity.com/paper/2311.05591/) (2023)

## Built on by (most cited first, 28 of 28)

- [MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents](https://mlsanity.com/paper/2501.14654/) (2025)
- [Sequential Diagnosis with Language Models](https://mlsanity.com/paper/2506.22405/) (2025)
- [ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room](https://mlsanity.com/paper/2505.22919/) (2025)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs](https://mlsanity.com/paper/2605.06915/) (2026)
- [Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimization](https://mlsanity.com/paper/2504.18080/) (2025)
- [Large Language Models as 'Hidden Persuaders': Fake Product Reviews are Indistinguishable to Humans and Machines](https://mlsanity.com/paper/2506.13313/) (2025)
- [Quantifying and Mitigating Premature Closure in Frontier LLMs](https://mlsanity.com/paper/2605.15000/) (2026)
- [Teaching large language models to reason like expert diagnosticians](https://mlsanity.com/paper/2509.12194/) (2025)
- [Evaluation of Causal Reasoning for Large Language Models in Contextualized Clinical Scenarios of Laboratory Test Interpretation](https://mlsanity.com/paper/2509.16372/) (2025)
- [PARC: An Autonomous Self-Reflective Coding Agent for Robust Execution of Long-Horizon Tasks](https://mlsanity.com/paper/2512.03549/) (2025)
- [Reverse Physician-AI Relationship: Full-process Clinical Diagnosis Driven by a Large Language Model](https://mlsanity.com/paper/2508.10492/) (2025)
- [How people use Copilot for Health](https://mlsanity.com/paper/2604.15331/) (2026)
- [InterFeat: A Pipeline for Finding Interesting Scientific Features](https://mlsanity.com/paper/2505.13534/) (2025)
- [Towards medical AI misalignment: a preliminary study](https://mlsanity.com/paper/2505.18212/) (2025)
- [Comparisons between a Large Language Model-based Real-Time Compound Diagnostic Medical AI Interface and Physicians for Common Internal Medicine Cases using Simulated Patients](https://mlsanity.com/paper/2505.20609/) (2025)
- [Treatment, evidence, imitation, and chat](https://mlsanity.com/paper/2506.23040/) (2025)
- [Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates](https://mlsanity.com/paper/2508.01159/) (2025)
- [Intuition emerges in Maximum Caliber models at criticality](https://mlsanity.com/paper/2508.06477/) (2025)
- [A global log for medical AI](https://mlsanity.com/paper/2510.04033/) (2025)
- [Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering](https://mlsanity.com/paper/2510.18691/) (2025)
- [Agentic AI Governance and Lifecycle Management in Healthcare](https://mlsanity.com/paper/2601.15630/) (2026)
- [Bridging AI and Clinical Reasoning: Abductive Explanations for Alignment on Critical Symptoms](https://mlsanity.com/paper/2602.13985/) (2026)
- [Eliciting associations between clinical variables from LLMs via comparison questions across populations](https://mlsanity.com/paper/2605.06335/) (2026)
- [Medical Model Synthesis Architectures: A Case Study](https://mlsanity.com/paper/2605.09716/) (2026)
- [Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher](https://mlsanity.com/paper/2606.01000/) (2026)
- [Six misconceptions about large language models: A minimal model and diagnostic taxonomy](https://mlsanity.com/paper/2608.20421/) (2026)

## Abstract (arXiv)

A seminal paper published by Ledley and Lusted in 1959 introduced complex clinical diagnostic reasoning cases as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. Here, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases against a baseline of hundreds of physicians. We conduct five experiments to measure clinical reasoning across differential diagnosis generation, display of diagnostic reasoning, triage differential diagnosis, probabilistic reasoning, and management reasoning, all adjudicated by physician experts with validated psychometrics. We then report a real-world study comparing human expert and AI second opinions in randomly-selected patients in the emergency room of a major tertiary academic medical center in Boston, MA. We compared LLMs and board-certified physicians at three predefined diagnostic touchpoints: triage in the emergency room, initial evaluation by a physician, and admission to the hospital or intensive care unit. In all experiments--both vignettes and emergency room second opinions--the LLM displayed superhuman diagnostic and reasoning abilities, as well as continued improvement from prior generations of AI clinical decision support. Our study suggests that LLMs have achieved superhuman performance on general medical diagnostic and management reasoning, fulfilling the vision put forth by Ledley and Lusted, and motivating the urgent need for prospective trials.