mlsanityData to 7 Oct 2026

Medical Vision Language Pretraining: A survey

arXiv 2312.06224 ↗First posted 11 Dec 2023cs.CV cs.CLcited by 24

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical Vision Language Pretraining (VLP) has recently emerged as a promising solution to the scarcity of labeled data in the medical domain. By leveraging paired/unpaired vision and text datasets through self-supervised learning, models can be trained to acquire vast knowledge and learn robust feature representations. Such pretrained models have the potential to enhance multiple downstream medical tasks simultaneously, reducing the dependency on labeled data. However, despite recent progress and its potential, there is no such comprehensive survey paper that has explored the various aspects and advancements in medical VLP. In this paper, we specifically review existing works through the lens of different pretraining objectives, architectures, downstream evaluation tasks, and datasets utilized for pretraining and downstream tasks. Subsequently, we delve into current challenges in medical VLP, discussing existing and potential solutions, and conclude by highlighting future directions. To the best of our knowledge, this is the first survey focused on medical VLP.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 114 of its 161 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

114 of 161 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training20 Oct
  2. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training11 Oct
  3. 2023Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images10 Oct
  4. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning12 Sep
  5. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision15 Aug
  6. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models15 Aug
  7. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization17 Jul
  8. 2023Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training14 Jul
  9. 2023Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark10 Jun
  10. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation7 Jun
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  12. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May
7 more
  1. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts18 May
  2. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training13 May
  3. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime30 Mar
  4. 2022Self-supervised vision-language pretraining for Medical visual question answering24 Nov
  5. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generation23 Nov
  6. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study30 Sep
  7. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May

Other references · 95

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2022Visual Prompt Tuningcited by 1,674
38 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  3. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2021Grounded Language-Image Pre-trainingcited by 1,147
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  7. 2019Publicly Available Clinical BERT Embeddingscited by 995
  8. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  9. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  11. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  12. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  13. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  14. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  15. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  16. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  17. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  18. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  19. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  20. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  21. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  22. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  23. 2021TransBTS: Multimodal Brain Tumor Segmentation Using Transformercited by 265
  24. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  25. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  26. 2023Improving CLIP Training with Language Rewritescited by 216
  27. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  28. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  29. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 198
  30. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 197
  31. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  32. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  33. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  34. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  35. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  36. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  37. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  38. 2020Intriguing Properties of Contrastive Lossescited by 129

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 11; 2026: 3.

24 papers.

Sort:

By area:

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  2. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  3. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  4. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learningcited by 16
  5. 2025Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Modelscited by 9
  6. 2025Multimodal Federated Learning With Missing Modalities through Feature Imputation Networkcited by 8
  7. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  8. 2023Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrievalcited by 6
  9. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rayscited by 6
  10. 2024CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalitiescited by 5
  11. 2025Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Modelscited by 5
  12. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classificationcited by 5
12 more
  1. 2024TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Modelscited by 3
  2. 20256 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Modelscited by 3
  3. 2025Knowledge to Sight: Reasoning over Visual Attributes via Knowledge Decomposition for Abnormality Groundingcited by 1
  4. 2025Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Diseasecited by 1
  5. 2025Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answeringcited by 1
  6. 2026Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcarecited by 1
  7. 2024SJTU:Spatial judgments in multimodal models towards unified segmentation through coordinate detectioncited by 0
  8. 2025CLIP-KOA: Enhancing Knee Osteoarthritis Diagnosis with Multi-Modal Learning and Symmetry-Aware Loss Functionscited by 0
  9. 2025Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Researchcited by 0
  10. 2025Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptationcited by 0
  11. 2026Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysiscited by 0
  12. 2026ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration
  2. 2022VLP: A Survey on Vision-Language Pre-training
  3. 2025Comprehensive language-image pre-training for 3D medical image understanding
  4. 2024Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
  5. 2024Medical Vision-Language Pre-Training for Brain Abnormalities
  6. 2024How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?
  7. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  8. 2024Uni-Mlip: Unified Self-supervision for Medical Vision Language Pre-training