mlsanityData to 7 Oct 2026

MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworks

arXiv 2503.07459 ↗First posted 10 Mar 2025cs.CL cs.AIcited by 38

On the map: LLM reasoning → Clinical and biomedical applications

also in 1 more
Abstract (arXiv)

Complex medical reasoning requires integrating heterogeneous clinical evidence across multiple inference steps. Large language models (LLMs) now approach this through two routes: internalized reasoning and externalized agent scaffolding (frameworks that decompose problems collaboratively amongst multiple LLMs). To determine whether these routes are exclusive or complementary, we introduce MedicalAgentsBench, a filtered benchmark of 862 complex clinical questions drawn from the union of eight medical datasets via difficulty-aware curation and contamination screening. Evaluating three internalized reasoning models (DeepSeek-R1, o1-mini, and o3-mini), seven base models, and nine externalized agent-based methods, we find that internalized and externalized approaches each independently improve performance, and that their benefits compound: the highest accuracy is achieved by layering agent workflows onto an internalized reasoning model (i.e., o3-mini + MDAgents with 35.1%). Pareto analysis shows this combination dominates the cost-performance frontier; moreover, lightweight optimization on inexpensive models offers an entry point for resource-constrained settings. Our benchmark is at https://github.com/gersteinlab/MedicalAgentsBench.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 38 of its 49 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

38 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MedCoT: Medical Chain of Thought via Hierarchical Expert18 Dec
  2. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoning5 May
  3. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning16 Nov
  4. 2022Can large language models reason about medical questions?17 Jul

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  11. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
22 more
  1. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  2. 2024AFlow: Automating Agentic Workflow Generationcited by 395
  3. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  4. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  5. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  6. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  7. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  8. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  9. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  10. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  11. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  12. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  13. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  14. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  15. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 100
  16. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 99
  17. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  18. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 58
  19. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  20. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  21. 2025Self-Supervised Prompt Optimizationcited by 37
  22. 2024AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Datasetcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 15.

38 papers.

Sort:

By area:

  1. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  4. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  5. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 34
  6. 2025MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Taskscited by 29
  7. 2025Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewardscited by 24
  8. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  9. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 21
  10. 2025MIRA: Multimodal Iterative Reasoning Agent for Image Editingcited by 15
  11. 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applicationscited by 10
  12. 2025HealthFlow: A Self-Evolving AI Agent with Meta Planning for Autonomous Healthcare Researchcited by 10
26 more
  1. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
  2. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systemscited by 8
  3. 2026Real Money, Fake Models: Deceptive Model Claims in Shadow APIscited by 8
  4. 2026MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMscited by 6
  5. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
  6. 2026GSEM: Graph-based Self-Evolving Memory for Experience Augmented Clinical Reasoningcited by 5
  7. 2025MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generationcited by 4
  8. 2025Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarkscited by 3
  9. 2025Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logscited by 3
  10. 2026ESTAR: Early-Stopping Token-Aware Reasoning For Efficient Inferencecited by 3
  11. 2026Medical thinking with multiple imagescited by 3
  12. 2025TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verificationcited by 2
  13. 2025Silence is Not Consensus: Disrupting Agreement Bias in Multi-Agent LLMs via Catfish Agent for Clinical Decision Makingcited by 2
  14. 2025ConfAgents: A Conformal-Guided Multi-Agent Framework for Cost-Efficient Medical Diagnosiscited by 2
  15. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  16. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  17. 2025MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Datacited by 1
  18. 2026MedCoG: Maximizing LLM Inference Density in Medical Reasoning via Meta-Cognitive Regulationcited by 1
  19. 2026Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learningcited by 1
  20. 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoningcited by 1
  21. 2025CURE: Confidence-driven Unified Reasoning Ensemble Framework for Medical Question Answeringcited by 0
  22. 2026MedCoRAG: Interpretable Hepatology Diagnosis via Hybrid Evidence Retrieval and Multispecialty Consensuscited by 0
  23. 2026From Exposure to Internalization: Dual-Stream Calibration for In-context Clinical Reasoningcited by 0
  24. 2026MultiDx: A Multi-Source Knowledge Integration Framework towards Diagnostic Reasoningcited by 0
  25. 2026Coupled Graph--Policy Distillation for Personalized Medication Safety in Older Adults with Multimorbiditycited by 0
  26. 2026Benchmarking Clinical Decision Pathway Adherence in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
  2. 2025ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning
  3. 2026A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing
  4. 2026MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation
  5. 2026From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAG
  6. 2024LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models
  7. 2025MedMMV: A Controllable Multimodal Multi-Agent Framework for Reliable and Verifiable Clinical Reasoning
  8. 2025Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge