mlsanityData to 7 Oct 2026

CLIP in Medical Imaging: A Survey

arXiv 2312.07353 ↗First posted 12 Dec 2023cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP), a simple yet effective pre-training paradigm, successfully introduces text supervision to vision models. It has shown promising results across various tasks due to its generalizability and interpretability. The use of CLIP has recently gained increasing interest in the medical imaging domain, serving as a pre-training paradigm for image-text alignment, or a critical component in diverse clinical tasks. With the aim of facilitating a deeper understanding of this promising direction, this survey offers an in-depth exploration of the CLIP within the domain of medical imaging, regarding both refined CLIP pre-training and CLIP-driven applications. In this paper, we (1) first start with a brief introduction to the fundamentals of CLIP methodology; (2) then investigate the adaptation of CLIP pre-training in the medical imaging domain, focusing on how to optimize CLIP given characteristics of medical images and reports; (3) further explore practical utilization of CLIP pre-trained models in various tasks, including classification, dense prediction, and cross-modal tasks; and (4) finally discuss existing limitations of CLIP in the context of medical imaging, and propose forward-looking directions to address the demands of medical imaging domain. Studies featuring technical and practical value are both investigated. We expect this survey will provide researchers with a holistic understanding of the CLIP paradigm and its potential implications. The project page of this survey can also be found on https://github.com/zhaozh10/Awesome-CLIP-in-Medical-Imaging.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 194 of its 319 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

194 of 319 references.

References · 194

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2017Graph Attention Networkscited by 8,854
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  13. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  14. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  15. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  16. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  17. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  18. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  19. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  20. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  21. 2022Language-driven Semantic Segmentationcited by 711
  22. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  23. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 597
  24. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  25. 2020Learning Continuous Image Representation with Local Implicit Image Functioncited by 550
  26. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  27. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  28. 2021Image Segmentation Using Text and Image Promptscited by 509
  29. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  30. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  31. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  32. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  33. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  34. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  35. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  36. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  37. 2016ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentationcited by 330
  38. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321

50 most cited of 194 All 194 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 22; 2026: 15.

40 papers.

Sort:

By area:

  1. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imagingcited by 20
  2. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  3. 2024μ-Bench: A Vision-Language Benchmark for Microscopy Understandingcited by 15
  4. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?cited by 10
  5. 2025MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understandingcited by 10
  6. 2025Brain-Adapter: Enhancing Neurological Disorder Analysis with Adapter-Tuning Multimodal Large Language Modelscited by 6
  7. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  8. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokenscited by 5
  9. 2025TextSAM-EUS: Text Prompt Learning for SAM to Accurately Segment Pancreatic Tumor in Endoscopic Ultrasoundcited by 5
  10. 2025Dual Cross-image Semantic Consistency with Self-aware Pseudo Labeling for Semi-supervised Medical Image Segmentationcited by 4
  11. 2025Multimodal Medical Image Binding via Shared Text Embeddingscited by 3
  12. 2025Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretationcited by 3
28 more
  1. 2026MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentationcited by 3
  2. 2025Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Modelscited by 2
  3. 2025Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approachescited by 2
  4. 2025Enabling Ultra-Fast Cardiovascular Imaging Across Heterogeneous Clinical Environments with A Generalist Foundation Model and Multimodal Databasecited by 2
  5. 2025Adapting Foundation Model for Dental Caries Detection with Dual-View Co-Trainingcited by 1
  6. 2025Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rayscited by 1
  7. 2025Revolutionizing Precise Low Back Pain Diagnosis via Contrastive Learningcited by 1
  8. 2025Surgeons Are Indian Males and Speech Therapists Are White Females: Auditing Biases in Vision-Language Models for Healthcare Professionalscited by 1
  9. 2025CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selectioncited by 1
  10. 2025Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imagingcited by 1
  11. 2025VL-OrdinalFormer: Vision Language Guided Ordinal Transformers for Interpretable Knee Osteoarthritis Gradingcited by 1
  12. 2025Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridgingcited by 0
  13. 2025Role-SynthCLIP: A Role Play Driven Diverse Synthetic Data Approachcited by 0
  14. 2025Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosiscited by 0
  15. 2026CytoCLIP: Learning Cytoarchitectural Characteristics in Developing Human Brain Using Contrastive Language Image Pre-Trainingcited by 0
  16. 2026CauCLIP: Bridging the Sim-to-Real Gap in Surgical Video Understanding via Causality-Inspired Vision-Language Modelingcited by 0
  17. 2026XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Modelscited by 0
  18. 2026Foundation Models for Medical Imaging: Status, Challenges, and Directionscited by 0
  19. 2026Towards Calibrating Prompt Tuning of Vision-Language Modelscited by 0
  20. 2026Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimationcited by 0
  21. 2026MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentationcited by 0
  22. 2026Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Modelscited by 0
  23. 2026Are Natural-Domain Foundation Models Effective for Accelerated Cardiac MRI Reconstruction?cited by 0
  24. 2026Toward Multimodal Conversational AI for Age-Related Macular Degenerationcited by 0
  25. 2026Learning from Acquisition: Metadata-driven Multimodal Pre-training for Cardiac MRIcited by 0
  26. 2026SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representationcited by 0
  27. 2026Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Modelscited by 0
  28. 2026Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Modelcited by 0