mlsanityData to 7 Oct 2026

MMedAgent: Learning to Use Medical Tools with Multi-modal Agent

arXiv 2407.02483 ↗First posted 2 Jul 2024cs.CL cs.AIcited by 108

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 1 more
Abstract (arXiv)

Multi-Modal Large Language Models (MLLMs), despite being successful, exhibit limited generality and often fall short when compared to specialized models. Recently, LLM-based agents have been developed to address these challenges by selecting appropriate specialized models as tools based on user inputs. However, such advancements have not been extensively explored within the medical domain. To bridge this gap, this paper introduces the first agent explicitly designed for the medical field, named Multi-modal Medical Agent (MMedAgent). We curate an instruction-tuning dataset comprising six medical tools solving seven tasks across five modalities, enabling the agent to choose the most suitable tools for a given task. Comprehensive experiments demonstrate that MMedAgent achieves superior performance across a variety of medical tasks compared to state-of-the-art open-source methods and even the closed-source model, GPT-4o. Furthermore, MMedAgent exhibits efficiency in updating and integrating new medical tools. Codes and models are all available.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 56 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

34 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  2. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology24 May

Other references · 32

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Segment Anythingcited by 8,158
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  11. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  12. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
20 more
  1. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  2. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  3. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  4. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  5. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  6. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  7. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  8. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  9. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
  10. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  11. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  12. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  13. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 107
  14. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  15. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  16. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  17. 2023LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editingcited by 47
  18. 2023The Multi-modality Cell Segmentation Challenge: Towards Universal Solutionscited by 39
  19. 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
  20. 2023Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 47; 2026: 50.

108 papers.

Sort:

By area:

  1. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 84
  2. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
  3. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  4. 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
  5. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 56
  6. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledgecited by 50
  7. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  8. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  9. 2024DrugAgent: Automating AI-aided Drug Discovery Programming through LLM Multi-Agent Collaborationcited by 40
  10. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  11. 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Frameworkcited by 33
  12. 2025Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agentscited by 33
38 more
  1. 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 25
  2. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  3. 2024ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agentscited by 20
  4. 2025Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agentscited by 20
  5. 2026ToolTree: Efficient LLM Agent Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional Pruningcited by 18
  6. 2025A co-evolving agentic AI system for medical imaging analysiscited by 16
  7. 2025SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysiscited by 15
  8. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 12
  9. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  10. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  11. 2025End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoningcited by 10
  12. 2026IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentationcited by 10
  13. 2026Evolving Medical Imaging Agents via Experience-driven Self-skill Discoverycited by 10
  14. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  15. 2025ADAgent: LLM Agent for Alzheimer's Disease Analysis with Collaborative Coordinatorcited by 9
  16. 2025MedLA: A Logic-Driven Multi-Agent Framework for Complex Medical Reasoning with Large Language Modelscited by 9
  17. 2025Emerging Cyber Attack Risks of Medical AI Agentscited by 8
  18. 2025RadAgents: Multimodal Agentic Reasoning for Chest X-ray Interpretation with Radiologist-like Workflowscited by 8
  19. 2025Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoningcited by 8
  20. 2025WSI-Agents: A Collaborative Multi-Agent System for Multi-Modal Whole Slide Image Analysiscited by 7
  21. 2025CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretationcited by 7
  22. 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
  23. 2024RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatmentcited by 6
  24. 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learningcited by 6
  25. 2026Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improvecited by 6
  26. 2024A Learnable Agent Collaboration Network Framework for Personalized Multimodal AI Search Enginecited by 5
  27. 2024From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuningcited by 5
  28. 2025Privacy-Enhancing Paradigms within Federated Multi-Agent Systemscited by 5
  29. 2025Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generationcited by 5
  30. 2025MRGAgents: A Multi-Agent Framework for Improved Medical Report Generation with Med-LVLMscited by 5
  31. 2025AURA: A Multi-Modal Medical Agent for Understanding, Reasoning & Annotationcited by 5
  32. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  33. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generationcited by 5
  34. 20263DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysiscited by 5
  35. 2026MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflowscited by 5
  36. 2025CollEX -- A Multimodal Agentic RAG System Enabling Interactive Exploration of Scientific Collectionscited by 4
  37. 2025Learning to Be A Doctor: Searching for Effective Medical Agent Architecturescited by 4
  38. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration
  2. 2026Meissa: Multi-modal Medical Agentic Intelligence
  3. 2025Multimodal Large Language Models for Medicine: A Comprehensive Survey
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
  5. 2024Inquire, Interact, and Integrate: A Proactive Agent Collaborative Framework for Zero-Shot Multimodal Medical Reasoning
  6. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
  7. 2026AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks
  8. 2026ArogyaSutra: A Multi-Agent Framework for Multimodal Medical Reasoning in Indic Languages