mlsanityData to 14 Oct 2026

AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

arXiv 2402.09742 ↗First posted 15 Feb 2024cs.CLcited by 101

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Artificial intelligence has significantly advanced healthcare, particularly through large language models (LLMs) that excel in medical question answering benchmarks. However, their real-world clinical application remains limited due to the complexities of doctor-patient interactions. To address this, we introduce AI Hospital, a multi-agent framework simulating dynamic medical interactions between Doctor as player and NPCs including Patient, Examiner, Chief Physician. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation (MVME) benchmark, utilizing high-quality Chinese medical records and NPCs to evaluate LLMs' performance in symptom collection, examination recommendations, and diagnoses. Additionally, a dispute resolution collaborative mechanism is proposed to enhance diagnostic accuracy through iterative discussions. Despite improvements, current LLMs exhibit significant performance gaps in multi-turn interactions compared to one-step approaches. Our findings highlight the need for further research to bridge these gaps and improve LLMs' clinical diagnostic capabilities. Our data, code, and experimental results are all open-sourced at urlhttps://github.com/LibertFan/AIHospital.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 59 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

30 of 59 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
18 more
  1. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  2. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  3. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 282
  4. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 162
  5. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 131
  6. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  7. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 120
  8. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  9. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 90
  10. 2022BioRED: A Rich Biomedical Relation Extraction Datasetcited by 85
  11. 2023BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPTcited by 65
  12. 2023DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuningcited by 64
  13. 2022A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasetscited by 45
  14. 2022MentSum: A Resource for Exploring Summarization of Mental Health Online Postscited by 11
  15. 2022DxFormer: A Decoupled Automatic Diagnostic System Based on Decoder-Encoder Transformer with Dense Symptom Representationscited by 10
  16. 2021Biomedical Data-to-Text Generation via Fine-Tuning Transformerscited by 9
  17. 2023KNSE: A Knowledge-aware Natural Language Inference Framework for Dialogue Symptom Status Recognitioncited by 5
  18. 2020Hierarchical Reinforcement Learning for Automatic Disease Diagnosiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 45; 2026: 48.

101 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 85
  4. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  5. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 47
  6. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  7. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 40
  8. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  9. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 40
  10. 2025PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactionscited by 32
  11. 2025Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learningcited by 27
  12. 2024ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaborationcited by 23
38 more
  1. 2025Multi-Agent System for Comprehensive Soccer Understandingcited by 18
  2. 2026Agent-as-a-Judgecited by 17
  3. 2024MedAide: Information Fusion and Anatomy of Medical Intents via LLM-based Agent Collaborationcited by 16
  4. 20253MDBench: Medical Multimodal Multi-agent Dialogue Benchmarkcited by 16
  5. 2025Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learningcited by 14
  6. 2024Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicinecited by 13
  7. 2025Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactionscited by 13
  8. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environmentscited by 13
  9. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Modelscited by 12
  10. 2026LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluationcited by 11
  11. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  12. 2025The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capabilitycited by 10
  13. 2025TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviewscited by 9
  14. 2025Adaptive-VP: A Framework for LLM-Based Virtual Patients that Adapts to Trainees' Dialogue to Facilitate Nurse Communication Trainingcited by 9
  15. 2024PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulationcited by 8
  16. 2025Many LLMs Are More Utilitarian Than Onecited by 8
  17. 2026HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agentscited by 8
  18. 2024From Intention To Implementation: Automating Biomedical Research via LLMscited by 7
  19. 2025ClinicalGPT-R1: Pushing reasoning capability of generalist disease diagnosis with large language modelcited by 7
  20. 2025ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Educationcited by 6
  21. 2025Empowering Medical Multi-Agents with Clinical Consultation Flow for Dynamic Diagnosiscited by 5
  22. 2025FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarioscited by 5
  23. 2025Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluationcited by 5
  24. 2026Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verificationcited by 5
  25. 2025ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recoverycited by 4
  26. 2025Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domaincited by 4
  27. 2025CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environmentcited by 4
  28. 2026Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correctioncited by 4
  29. 2026MARCH: Multi-Agent Radiology Clinical Hierarchy for CT Report Generationcited by 4
  30. 2026MeDxAgent: Multi-Agent Consultation for Interactive Medical Diagnosiscited by 4
  31. 2024AI-Press: A Multi-Agent News Generating and Feedback Simulation System Powered by Large Language Modelscited by 3
  32. 2025Fine-tuning Vision Language Models with Graph-based Knowledge for Explainable Medical Image Analysiscited by 3
  33. 2026Strong Reasoning Isn't Enough: Evaluating Evidence Elicitation in Interactive Diagnosiscited by 3
  34. 2026RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agentscited by 3
  35. 2026MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environmentscited by 3
  36. 2026MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical Interactioncited by 3
  37. 2025LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluationcited by 2
  38. 2025MedSyn: Enhancing Diagnostics with Human-AI Collaborationcited by 2

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
  2. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
  3. 2024Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education
  4. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
  5. 2025Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks
  6. 2024Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios
  7. 2025MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
  8. 2025Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversations