mlsanityData to 14 Oct 2026

3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark

arXiv 2504.13861 ↗First posted 26 Mar 2025cs.HC cs.CL cs.MAcited by 16

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 1 more
Abstract (arXiv)

Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct complex real-world telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. This paper presents 3MDBench (Medical Multimodal Multi-agent Dialogue Benchmark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through temperament-based Patient Agent and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. It includes 2996 cases across 34 diagnoses from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for widely used open and closed-source LVLMs. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional neural network into the LVLM's context boosts F1 by up to 20%. Source code is available at https://github.com/univanxx/3mdbench.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 110 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

38 of 110 references.

References · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  12. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
26 more
  1. 2023The Falcon Series of Open Language Modelscited by 535
  2. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  3. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  4. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  5. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  6. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 367
  7. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  8. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  9. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  10. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 185
  11. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  12. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 119
  13. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 101
  14. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 60
  15. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  16. 2022A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasetscited by 45
  17. 2023CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcarecited by 43
  18. 2024K-QA: A Real-World Medical Q&A Benchmarkcited by 31
  19. 2023Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Datacited by 26
  20. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  21. 2024Crowdsourcing Dermatology Images with Google Search Ads: Creating a Real-World Skin Condition Datasetcited by 21
  22. 2024MedSyn: LLM-based Synthetic Medical Text Generation Frameworkcited by 19
  23. 2025Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoningcited by 19
  24. 2023LLM-Mini-CEX: Automatic Evaluation of Large Language Model for Diagnostic Conversationcited by 14
  25. 2024PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluationcited by 10
  26. 2024An Empirical Evaluation of Large Language Models on Consumer Health Questionscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Modelscited by 12
  3. 2025Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safetycited by 7
  4. 2026MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical Interactioncited by 3
  5. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  6. 2025MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Datacited by 2
  7. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  8. 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoningcited by 1
  9. 2026MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounterscited by 1
  10. 2025Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applicationscited by 0
  11. 2026LLM-Guided Evolution for Medical Decision Pipelinescited by 0
  12. 2026IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversationscited by 0
4 more
  1. 2026MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultationcited by 0
  2. 2026ELICITED: EHR-grounded Longitudinal Interactive Conversations for Information-seeking Triage Evaluation and Decision-makingcited by 0
  3. 2026A Source-Grounded Framework for Constructing and Evaluating Progressive Multimodal Diagnostic Dialogues from Clinical Case Reportscited by 0
  4. 2026MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Conversational Diagnostic AI with Multimodal Reasoning
  2. 2026PatientVLM Meets DocVLM: Pre-Consultation Dialogue Between Vision-Language Models for Efficient Diagnosis
  3. 2025MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
  4. 2026MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems
  5. 2025Architecting Clinical Collaboration: Multi-Agent Reasoning Systems for Multimodal Medical VQA
  6. 2026PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
  7. 2026ArogyaSutra: A Multi-Agent Framework for Multimodal Medical Reasoning in Indic Languages
  8. 2025MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration