# Towards Accurate Differential Diagnosis with Large Language Models

arXiv: https://arxiv.org/abs/2312.00164 · HTML: https://mlsanity.com/paper/2312.00164/

## Key facts

- Cited by 116 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 43 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://mlsanity.com/paper/2307.15343/) (2023)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations](https://mlsanity.com/paper/2311.05584/) (2023)
- [Generative Relevance Feedback with Large Language Models](https://mlsanity.com/paper/2304.13157/) (2023)
- [Multimodal Foundation Models Exploit Text to Make Medical Image Predictions](https://mlsanity.com/paper/2311.05591/) (2023)
- [Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task](https://mlsanity.com/paper/2211.05039/) (2022)

## Built on by (most cited first, 30 of 116)

- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making](https://mlsanity.com/paper/2404.15155/) (2024)
- [Towards a Science of Scaling Agent Systems](https://mlsanity.com/paper/2512.08296/) (2025)
- [Towards Conversational Diagnostic AI](https://mlsanity.com/paper/2401.05654/) (2024)
- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [Sequential Diagnosis with Language Models](https://mlsanity.com/paper/2506.22405/) (2025)
- [The opportunities and risks of large language models in mental health](https://mlsanity.com/paper/2403.14814/) (2024)
- [An Agentic System for Rare Disease Diagnosis with Traceable Reasoning](https://mlsanity.com/paper/2506.20430/) (2025)
- [SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data](https://mlsanity.com/paper/2505.20347/) (2025)
- [WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image](https://mlsanity.com/paper/2412.02141/) (2024)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025)
- [MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports](https://mlsanity.com/paper/2505.11733/) (2025)
- [Superhuman performance of a large language model on the reasoning tasks of a physician](https://mlsanity.com/paper/2412.10849/) (2024)
- [A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains](https://mlsanity.com/paper/2507.23486/) (2025)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [STEER: Assessing the Economic Rationality of Large Language Models](https://mlsanity.com/paper/2402.09552/) (2024)
- [Towards a Personal Health Large Language Model](https://mlsanity.com/paper/2406.06474/) (2024)
- [What Are the Odds? Language Models Are Capable of Probabilistic Reasoning](https://mlsanity.com/paper/2406.12830/) (2024)
- [MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications](https://mlsanity.com/paper/2409.07314/) (2024)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [Transforming Wearable Data into Personal Health Insights using Large Language Model Agents](https://mlsanity.com/paper/2406.06464/) (2024)
- [Working with AI: Measuring the Applicability of Generative AI to Occupations](https://mlsanity.com/paper/2507.07935/) (2025)
- [Language Agents for Hypothesis-driven Clinical Decision Making with Reinforcement Learning](https://mlsanity.com/paper/2506.13474/) (2025)
- [InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training](https://mlsanity.com/paper/2510.15859/) (2025)
- [Standardization of Psychiatric Diagnoses -- Role of Fine-tuned LLM Consortium and OpenAI-gpt-oss Reasoning LLM Enabled Decision Support System](https://mlsanity.com/paper/2510.25588/) (2025)
- [A Scalable Framework for Evaluating Health Language Models](https://mlsanity.com/paper/2503.23339/) (2025)
- [Proof-of-TBI -- Fine-Tuned Vision Language Model Consortium and OpenAI-o3 Reasoning LLM-Based Medical Diagnosis Support System for Mild Traumatic Brain Injury (TBI) Prediction](https://mlsanity.com/paper/2504.18671/) (2025)

## Abstract (arXiv)

An accurate differential diagnosis (DDx) is a cornerstone of medical care, often reached through an iterative process of interpretation that combines clinical history, physical examination, investigations and procedures. Interactive interfaces powered by Large Language Models (LLMs) present new opportunities to both assist and automate aspects of this process. In this study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its ability to generate a DDx alone or as an aid to clinicians. 20 clinicians evaluated 302 challenging, real-world medical cases sourced from the New England Journal of Medicine (NEJM) case reports. Each case report was read by two clinicians, who were randomized to one of two assistive conditions: either assistance from search engines and standard medical resources, or LLM assistance in addition to these tools. All clinicians provided a baseline, unassisted DDx prior to using the respective assistive tools. Our LLM for DDx exhibited standalone performance that exceeded that of unassisted clinicians (top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study arms, the DDx quality score was higher for clinicians assisted by our LLM (top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%) (McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p = 0.03). Further, clinicians assisted by our LLM arrived at more comprehensive differential lists than those without its assistance. Our study suggests that our LLM for DDx has potential to improve clinicians' diagnostic reasoning and accuracy in challenging cases, meriting further real-world evaluation for its ability to empower physicians and widen patients' access to specialist-level expertise.