mlsanityData to 14 Oct 2026

Vision Language Models in Medicine

arXiv 2503.01863 ↗First posted 24 Feb 2025cs.CV cs.AI cs.CLcited by 12

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

With the advent of Vision-Language Models (VLMs), medical artificial intelligence (AI) has experienced significant technological progress and paradigm shifts. This survey provides an extensive review of recent advancements in Medical Vision-Language Models (Med-VLMs), which integrate visual and textual data to enhance healthcare outcomes. We discuss the foundational technology behind Med-VLMs, illustrating how general models are adapted for complex medical tasks, and examine their applications in healthcare. The transformative impact of Med-VLMs on clinical practice, education, and patient care is highlighted, alongside challenges such as data scarcity, narrow task generalization, interpretability issues, and ethical concerns like fairness, accountability, and privacy. These limitations are exacerbated by uneven dataset distribution, computational demands, and regulatory hurdles. Rigorous evaluation methods and robust regulatory frameworks are essential for safe integration into healthcare workflows. Future directions include leveraging large-scale, diverse datasets, improving cross-modal generalization, and enhancing interpretability. Innovations like federated learning, lightweight architectures, and Electronic Health Record (EHR) integration are explored as pathways to democratize access and improve clinical relevance. This review aims to provide a comprehensive understanding of Med-VLMs' strengths and limitations, fostering their ethical and balanced adoption in healthcare.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 76 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

60 of 76 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicine16 Oct
  2. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI6 Aug
  3. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  4. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  6. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May
  7. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Segment Anythingcited by 8,234
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  3. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  4. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  11. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  13. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  14. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  15. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  16. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  17. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  18. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  19. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  20. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  21. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  22. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  23. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 319
  24. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  25. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  26. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
  27. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  28. 2024An Introduction to Vision-Language Modelingcited by 178
  29. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  30. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 106
  31. 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
  32. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 80
  33. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 55
  34. 2020FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrievalcited by 55
  35. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  36. 2024CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generationcited by 25
  37. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  38. 2024Hallucination Benchmark in Medical Visual Question Answeringcited by 18

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Surveycited by 7
  2. 2026MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learningcited by 5
  3. 2025Robust Fairness Vision-Language Learning for Medical Image Analysiscited by 4
  4. 2026AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Taskscited by 3
  5. 2025Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AIcited by 2
  6. 2026CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generationcited by 1
  7. 2026BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routinecited by 1
  8. 2025PediatricsMQA: a Multi-modal Pediatrics Question Answering Benchmarkcited by 0
  9. 2026Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Modelscited by 0
  10. 2026FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language modelcited by 0
  11. 2026Multilingual Hematology Visual Question Answering Datasetcited by 0
  12. 2026An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  2. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
  4. 2025Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insights
  5. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge
  6. 2025Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
  7. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
  8. 2026Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?