mlsanityData to 14 Oct 2026

Multimodal Large Language Models for Medicine: A Comprehensive Survey

arXiv 2504.21051 ↗First posted 29 Apr 2025cs.LG cs.CL cs.MMcited by 16

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

MLLMs have recently become a focal point in the field of artificial intelligence research. Building on the strong capabilities of LLMs, MLLMs are adept at addressing complex multi-modal tasks. With the release of GPT-4, MLLMs have gained substantial attention from different domains. Researchers have begun to explore the potential of MLLMs in the medical and healthcare domain. In this paper, we first introduce the background and fundamental concepts related to LLMs and MLLMs, while emphasizing the working principles of MLLMs. Subsequently, we summarize three main directions of application within healthcare: medical reporting, medical diagnosis, and medical treatment. Our findings are based on a comprehensive review of 330 recent papers in this area. We illustrate the remarkable capabilities of MLLMs in these domains by providing specific examples. For data, we present six mainstream modes of data along with their corresponding evaluation benchmarks. At the end of the survey, we discuss the challenges faced by MLLMs in the medical and healthcare domain and propose feasible methods to mitigate or overcome these issues.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 185 of its 300 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

185 of 300 references.

Closest ancestors · 25

Same direction, newest first

  1. 2024MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis19 Oct
  2. 2024RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction7 Oct
  3. 2024MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation29 Sep
  4. 2024OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography30 Aug
  5. 2024M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation29 Aug
  6. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenario16 Aug
  7. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning15 Aug
  8. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  9. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports8 Jul
  10. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale27 Jun
  11. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine14 May
  12. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models16 Apr
13 more
  1. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
  2. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  3. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging5 Jan
  4. 2023MAIRA-1: A specialised large multimodal model for radiology report generation22 Nov
  5. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare27 Oct
  6. 2023CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images22 Oct
  7. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training11 Oct
  8. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue21 Jun
  9. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models13 Jun
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  11. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology24 May
  12. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts18 May
  13. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains9 Oct

Other references · 160

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2024The Llama 3 Herd of Modelscited by 14,715
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  11. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  12. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
38 more
  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2023Segment Anythingcited by 8,234
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  13. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  14. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  16. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  17. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  18. 2020Extracting Training Data from Large Language Modelscited by 2,189
  19. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  20. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  21. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  22. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  23. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  24. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  25. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  26. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  27. 2023PaLM 2 Technical Reportcited by 1,162
  28. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  29. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  30. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  31. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  32. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  33. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  34. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  35. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  36. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  37. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  38. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475

50 most cited of 160 All 160 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  2. 2025OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Taskscited by 8
  3. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  4. 2026Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Benchcited by 3
  5. 2025Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiologycited by 2
  6. 2025Human or LLM as Standardized Patients? A Comparative Study for Medical Educationcited by 2
  7. 2025From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validationcited by 2
  8. 2026Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Trainingcited by 2
  9. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  10. 2025SpiroLLM: Finetuning Pretrained LLMs to Understand Spirogram Time Series with Clinical Validation in COPD Reportingcited by 1
  11. 2025TomoGraphView: 3D Medical Image Classification with Omnidirectional Slice Representations and Graph Neural Networkscited by 1
  12. 2026BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routinecited by 1
4 more
  1. 2025Generating Findings for Jaw Cysts in Dental Panoramic Radiographs Using a GPT-Based VLM: A Preliminary Study on Building a Two-Stage Self-Correction Loop with Structured Output (SLSO) Frameworkcited by 0
  2. 2026Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Modelscited by 0
  3. 2026Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMscited by 0
  4. 2026MULTISEISMO: A Multimodal Seismic Dataset and Model for Cross-Modal Seismic Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
  2. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  3. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
  4. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
  5. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  7. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  8. 2025Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks