mlsanityData to 14 Oct 2026

Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study

arXiv 2209.15517 ↗First posted 30 Sep 2022cs.CVcited by 66

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The large-scale pre-trained vision language models (VLM) have shown remarkable domain transfer capability on natural images. However, it remains unknown whether this capability can also apply to the medical image domain. This paper thoroughly studies the knowledge transferability of pre-trained VLMs to the medical domain, where we show that well-designed medical prompts are the key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting with expressive attributes that are shared between domains, the VLM can carry the knowledge across domains and improve its generalization. This mechanism empowers VLMs to recognize novel objects with fewer or without image samples. Furthermore, to avoid the laborious manual designing process, we develop three approaches for automatic generation of medical prompts, which can inject expert-level medical knowledge and image-specific information into the prompts for fine-grained grounding. We conduct extensive experiments on thirteen different medical datasets across various modalities, showing that our well-designed prompts greatly improve the zero-shot performance compared to the default prompts, and our fine-tuned models surpass the supervised models by a significant margin.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 53 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

39 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  6. 2015Fast R-CNNcited by 2,933
  7. 2018YOLOv3: An Incremental Improvementcited by 2,659
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  12. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
26 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2016YOLO9000: Better, Faster, Strongercited by 1,431
  3. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  5. 2021Grounded Language-Image Pre-trainingcited by 1,156
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  7. 2019How Can We Know What Language Models Know?cited by 959
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  9. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  10. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 589
  11. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  12. 2020PraNet: Parallel Reverse Attention Network for Polyp Segmentationcited by 460
  13. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  14. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  15. 2016A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Imagescited by 221
  16. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 201
  17. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 198
  18. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  19. 2021Dynamic Head: Unifying Object Detection Heads with Attentionscited by 187
  20. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  21. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  22. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  23. 2018Methods for Segmentation and Classification of Digital Microscopy Tissue Imagescited by 73
  24. 2021Supervised Transfer Learning at Scale for Medical Imagingcited by 38
  25. 2020Deep Learning in Diabetic Foot Ulcers Detection: A Comprehensive Evaluationcited by 19
  26. 2020Distant Domain Transfer Learning for Medical Imagingcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 23; 2025: 17; 2026: 4.

66 papers.

Sort:

By area:

  1. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 271
  2. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  3. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
  4. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
  5. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  6. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  7. 2023Continual Learning for Abdominal Multi-Organ and Tumor Segmentationcited by 43
  8. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  9. 2023MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classificationcited by 38
  10. 2023Medical Phrase Grounding with Region-Phrase Context Contrastive Alignmentcited by 32
  11. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  12. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
38 more
  1. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  2. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Modelscited by 28
  3. 2023Medical Vision Language Pretraining: A surveycited by 24
  4. 2024MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learningcited by 23
  5. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 18
  6. 2023Towards General Purpose Medical AI: Continual Learning Medical Foundation Modelcited by 14
  7. 2024DEAL: Disentangle and Localize Concept-level Explanations for VLMscited by 14
  8. 2024Frontiers in Intelligent Colonoscopycited by 14
  9. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  10. 2024Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-trainingcited by 13
  11. 2025UMIT: Unifying Medical Imaging Tasks via Vision-Language Modelscited by 12
  12. 2023Unleashing the Potential of Large Language Model: Zero-shot VQA for Flood Disaster Scenariocited by 10
  13. 2025DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?cited by 10
  14. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  15. 2023Deblurring Masked Autoencoder is Better Recipe for Ultrasound Image Recognitioncited by 8
  16. 2024Aligning Medical Images with General Knowledge from Large Language Modelscited by 8
  17. 2025MedFILIP: Medical Fine-grained Language-Image Pre-trainingcited by 8
  18. 2024Disease-informed Adaptation of Vision-Language Modelscited by 7
  19. 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Modelscited by 7
  20. 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variationscited by 7
  21. 2023Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiographycited by 6
  22. 2024Freeze the backbones: A Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-trainingcited by 6
  23. 2024Beyond Accuracy: Ensuring Correct Predictions With Correct Rationalescited by 6
  24. 2025Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Modelscited by 6
  25. 2026MRAD: Zero-Shot Anomaly Detection with Memory-Driven Retrievalcited by 6
  26. 2023Few-shot medical image classification with simple shape and texture text descriptors using vision-language modelscited by 5
  27. 2024TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotationcited by 5
  28. 2024OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicinecited by 5
  29. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 5
  30. 2023SegPrompt: Using Segmentation Map as a Better Prompt to Finetune Deep Models for Kidney Stone Classificationcited by 4
  31. 2023LAMP: Leveraging Language Prompts for Multi-person Pose Estimationcited by 4
  32. 2024Enhancing Gait Video Analysis in Neurodegenerative Diseases by Knowledge Augmentation in Vision Language Modelcited by 4
  33. 2024Dual-Image Enhanced CLIP for Zero-Shot Anomaly Detectioncited by 4
  34. 2025Text-Promptable Propagation for Referring Medical Image Sequence Segmentationcited by 4
  35. 2024AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained Modelscited by 3
  36. 2025Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detectioncited by 3
  37. 2025iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detectioncited by 3
  38. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Modelscited by 3

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classification
  2. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  3. 2025Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models
  4. 2024Medical Vision-Language Pre-Training for Brain Abnormalities
  5. 2026Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
  6. 2024VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge
  7. 2025Comprehensive language-image pre-training for 3D medical image understanding
  8. 2024Exploring Low-Resource Medical Image Classification with Weakly Supervised Prompt Learning