mlsanityData to 14 Oct 2026

Automating Expert-Level Medical Reasoning Evaluation of Large Language Models

arXiv 2507.07988 ↗First posted 10 Jul 2025cs.CLcited by 22

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring transparent and trustworthy reasoning is essential. However, existing evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains lacking. To address this, we introduce MedThink-Bench, a benchmark designed for rigorous, explainable, and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 challenging questions across ten medical domains, each annotated with expert-crafted step-by-step rationales. Building on this, we propose LLM-w-Ref, a novel evaluation framework that leverages fine-grained rationales and LLM-as-a-Judge mechanisms to assess intermediate reasoning with expert-level fidelity while maintaining scalability. Experiments show that LLM-w-Ref exhibits a strong positive correlation with expert judgments. Benchmarking twelve state-of-the-art LLMs, we find that smaller models (e.g., MedGemma-27B) can surpass larger proprietary counterparts (e.g., OpenAI-o3). Overall, MedThink-Bench offers a foundational tool for evaluating LLMs' medical reasoning, advancing their safe and responsible deployment in clinical practice.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 48 of its 81 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

48 of 81 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models20 May
  2. 2025MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports16 May
  3. 2025Building a Human-Verified Clinical Reasoning Dataset via a Human LLM Hybrid Pipeline for Trustworthy Medical AI11 May
  4. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworks10 Mar
  5. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases6 Mar
  6. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs25 Dec
  7. 2024Superhuman performance of a large language model on the reasoning tasks of a physician14 Dec
  8. 2023Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine13 Aug
  9. 2022Can large language models reason about medical questions?17 Jul

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  11. 2024A Survey on LLM-as-a-Judgecited by 1,423
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
27 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  4. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  5. 2025Humanity's Last Examcited by 491
  6. 2025MedGemma Technical Reportcited by 436
  7. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
  8. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 214
  9. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
  10. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  11. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  12. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 109
  13. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  14. 2024Med42-v2: A Suite of Clinical LLMscited by 90
  15. 2022DDXPlus: A New Dataset For Automatic Medical Diagnosiscited by 88
  16. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  17. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 60
  18. 2024Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approachescited by 52
  19. 2024The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)cited by 45
  20. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  21. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  22. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
  23. 2024Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithmcited by 14
  24. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  25. 2023ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examinationcited by 10
  26. 2025Uncertainty-Aware Large Language Models for Explainable Disease Diagnosiscited by 9
  27. 2025Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgmentscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 19.

22 papers.

Sort:

By area:

  1. 2025Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatchcited by 14
  2. 2025MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Informationcited by 8
  3. 2025Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendationcited by 3
  4. 2026Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Studycited by 3
  5. 2026EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMscited by 2
  6. 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoningcited by 2
  7. 2026MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounterscited by 1
  8. 2026Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AIcited by 1
  9. 2026To Reason or Not to: Selective Chain-of-Thought in Medical Question Answeringcited by 0
  10. 2026Rethinking Patient Education as Multi-turn Multi-modal Interactioncited by 0
  11. 2026VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphscited by 0
  12. 2026Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcarecited by 0
10 more
  1. 2026GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibrationcited by 0
  2. 2026Toward Vibe Medicine: A Self-Evolving Multi-Agent Framework for Clinical Decision Supportcited by 0
  3. 2026MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Modelscited by 0
  4. 2026CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarioscited by 0
  5. 2026RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Carecited by 0
  6. 2026A primer on evaluation methods for large language models in healthcarecited by 0
  7. 2026Large language models in medical time series analysiscited by 0
  8. 2026CRISS: A Retrieval-Augmented AI Chatbot for Assisting Cancer Registrarscited by 0
  9. 2026BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learningcited by 0
  10. 2026A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combinedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine
  2. 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
  3. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
  4. 2025Disentangling Reasoning and Knowledge in Medical Large Language Models
  5. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
  6. 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
  7. 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications
  8. 2024Critique of Impure Reason: Unveiling the reasoning behaviour of medical Large Language Models