mlsanityData to 7 Oct 2026

AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

arXiv 2405.07960 ↗First posted 13 May 2024cs.HC cs.CLcited by 215

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages. We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs' ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore novel patient-centric metrics that this interactive environment firstly enables.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 37 of its 60 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

37 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents5 May

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
24 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  5. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  8. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  9. 2023Active Retrieval Augmented Generationcited by 506
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  12. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  13. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  14. 2022Can large language models reason about medical questions?cited by 194
  15. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  16. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  17. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
  18. 2024Towards Conversational Diagnostic AIcited by 79
  19. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  20. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  21. 2024Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulatorcited by 36
  22. 2024Addressing cognitive bias in medical language modelscited by 34
  23. 2024Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answeringcited by 31
  24. 2023Language models are susceptible to incorrect patient self-diagnosis in medical applicationscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 86; 2026: 110.

215 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  2. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  4. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  5. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  6. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 84
  7. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  8. 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
  9. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  10. 2024Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theorycited by 60
  11. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 59
  12. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 56
38 more
  1. 2025Sequential Diagnosis with Language Modelscited by 53
  2. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  3. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  5. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  6. 2024Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactionscited by 43
  7. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  8. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  9. 2025UXAgent: An LLM Agent-Based Usability Testing Framework for Web Designcited by 39
  10. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  11. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  12. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  13. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  14. 2025PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactionscited by 30
  15. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
  16. 2025CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logiccited by 27
  17. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  18. 2025Evidence-based diagnostic reasoning with multi-agent copilot for human pathologycited by 26
  19. 2025MDTeamGPT: A Self-Evolving LLM-based Multi-Agent Framework for Multi-Disciplinary Team Medical Consultationcited by 25
  20. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 24
  21. 2025Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewardscited by 24
  22. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 23
  23. 2025MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosiscited by 23
  24. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  25. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  26. 2025MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Frameworkcited by 22
  27. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  28. 2025Advancing Conversational Diagnostic AI with Multimodal Reasoningcited by 20
  29. 2025Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoningcited by 19
  30. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planningcited by 19
  31. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environmentscited by 19
  32. 2024GP-VLS: A general-purpose vision language model for surgerycited by 16
  33. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  34. 20253MDBench: Medical Multimodal Multi-agent Dialogue Benchmarkcited by 16
  35. 2024Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequencecited by 15
  36. 2025LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?cited by 15
  37. 2024MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedbackcited by 14
  38. 2024Enhancing Investment Analysis: Optimizing AI-Agent Collaboration in Financial Researchcited by 14

50 most cited of 215 All 215 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator
  2. 2026Agentic AI in medicine: architectures, applications, evaluation, and challenges for clinical translation
  3. 2026AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks
  4. 2025MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
  5. 2026MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems
  6. 2026A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing
  7. 2024ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents
  8. 2026Human-Guided Agentic AI for Multimodal Clinical Prediction: Lessons from the AgentDS Healthcare Benchmark