# Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases

arXiv: https://arxiv.org/abs/2503.04691 · HTML: https://mlsanity.com/paper/2503.04691/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 11 of 11)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [Baichuan-M1: Pushing the Medical Capability of Large Language Models](https://mlsanity.com/paper/2502.12671/) (2025)
- [Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator](https://mlsanity.com/paper/2403.08495/) (2024)
- [A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?](https://mlsanity.com/paper/2409.15277/) (2024)
- [Radiology-Aware Model-Based Evaluation Metric for Report Generation](https://mlsanity.com/paper/2311.16764/) (2023)
- [Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare](https://mlsanity.com/paper/2502.16051/) (2025)

## Built on by (most cited first, 29 of 29)

- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models](https://mlsanity.com/paper/2505.14107/) (2025)
- [EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis](https://mlsanity.com/paper/2510.25628/) (2025)
- [Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations](https://mlsanity.com/paper/2510.11196/) (2025)
- [Evolving Interactive Diagnostic Agents in a Virtual Clinical Environment](https://mlsanity.com/paper/2510.24654/) (2025)
- [CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection](https://mlsanity.com/paper/2506.05243/) (2025)
- [GSEM: Graph-based Self-Evolving Memory for Experience Augmented Clinical Reasoning](https://mlsanity.com/paper/2603.22096/) (2026)
- [MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation](https://mlsanity.com/paper/2503.17900/) (2025)
- [CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environment](https://mlsanity.com/paper/2512.10206/) (2025)
- [Med-REFL: Medical Reasoning Enhancement via Self-Corrected Fine-grained Reflection](https://mlsanity.com/paper/2506.13793/) (2025)
- [The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios](https://mlsanity.com/paper/2601.08173/) (2026)
- [Evaluating Hierarchical Clinical Document Classification Using Reasoning-Based LLMs](https://mlsanity.com/paper/2507.03001/) (2025)
- [An Empirical Study of Reasoning Steps in Thinking Code LLMs](https://mlsanity.com/paper/2511.05874/) (2025)
- [Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks](https://mlsanity.com/paper/2605.01417/) (2026)
- [EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs](https://mlsanity.com/paper/2605.30637/) (2026)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [Reverse Physician-AI Relationship: Full-process Clinical Diagnosis Driven by a Large Language Model](https://mlsanity.com/paper/2508.10492/) (2025)
- [Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning](https://mlsanity.com/paper/2607.07761/) (2026)
- [WiNGPT-3.0 Technical Report](https://mlsanity.com/paper/2505.17387/) (2025)
- [ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning](https://mlsanity.com/paper/2512.23440/) (2025)
- ["Do I Trust the AI?" Towards Trustworthy AI-Assisted Diagnosis: Understanding User Perception in LLM-Supported Reasoning](https://mlsanity.com/paper/2601.19540/) (2026)
- [Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients](https://mlsanity.com/paper/2604.01924/) (2026)
- [When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering](https://mlsanity.com/paper/2605.21807/) (2026)
- [CAREAgent: Clinical Agent with Structured Reasoning and Tool-Integrated for Order Generation](https://mlsanity.com/paper/2606.01094/) (2026)
- [ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models](https://mlsanity.com/paper/2606.03157/) (2026)
- [DEEPMED Search: An Open-Source Agentic Platform for Medical Deep Research with Introspective Verification](https://mlsanity.com/paper/2606.29746/) (2026)
- [Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs](https://mlsanity.com/paper/2607.18086/) (2026)
- [Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark](https://mlsanity.com/paper/2609.30027/) (2026)
- [From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning](https://mlsanity.com/paper/2609.39566/) (2026)

## Abstract (arXiv)

Recent advancements in reasoning-enhanced large language models (LLMs), such as DeepSeek-R1 and OpenAI-o3, have demonstrated significant progress. However, their application in professional medical contexts remains underexplored, particularly in evaluating the quality of their reasoning processes alongside final outputs. Here, we introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references derived from clinical case reports. Spanning 13 body systems and 10 specialties, it includes both common and rare diseases. To comprehensively evaluate LLM performance, we propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey. To assess reasoning quality, we present the Reasoning Evaluator, a novel automated system that objectively scores free-text reasoning responses based on efficiency, actuality, and completeness using dynamic cross-referencing and evidence checks. Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc. Our results show that current LLMs achieve over 85% accuracy in relatively simple diagnostic tasks when provided with sufficient examination results. However, performance declines in more complex tasks, such as examination recommendation and treatment planning. While reasoning outputs are generally reliable, with factuality scores exceeding 90%, critical reasoning steps are frequently missed. These findings underscore both the progress and limitations of clinical LLMs. Notably, open-source models like DeepSeek-R1 are narrowing the gap with proprietary systems, highlighting their potential to drive accessible and equitable advancements in healthcare.