mlsanityData to 14 Oct 2026

MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

arXiv 2210.10163 ↗First posted 18 Oct 2022cs.CV cs.CLcited by 570

Not on the map. Why?

Cited by 172 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing vision-text contrastive learning like CLIP aims to match the paired image and caption embeddings while pushing others apart, which improves representation transferability and supports zero-shot prediction. However, medical image-text datasets are orders of magnitude below the general images and captions from the internet. Moreover, previous methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this paper, we decouple images and texts for multimodal contrastive learning thus scaling the usable training data in a combinatorial magnitude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using around 200K data). Our code is available at https://github.com/RyanWangZf/MedCLIP.

Key facts. Cited by 570 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

34 of 46 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2020Supervised Contrastive Learningcited by 2,829
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  12. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
22 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  5. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  6. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  8. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  9. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  10. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  11. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  12. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 137
  13. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  14. 2020Learning Visual Representations with Caption Annotationscited by 123
  15. 2016Learning Visual N-Grams from Web Datacited by 95
  16. 2021Deep Learning for Chest X-ray Analysis: A Surveycited by 84
  17. 2021Weakly Supervised Contrastive Learningcited by 81
  18. 2017NegBio: a high-performance tool for negation and uncertainty detection in radiology reportscited by 79
  19. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  20. 2019Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representationscited by 36
  21. 2019Less Is Better: Unweighted Data Subsampling via Influence Functioncited by 30
  22. 2021Self-supervised Image-text Pre-training With Mixed Data In Chest X-rayscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 60; 2024: 151; 2025: 202; 2026: 154.

570 papers.

Sort:

By area:

  1. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  2. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  3. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  4. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  5. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 134
  6. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  7. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  8. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
  9. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  10. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  11. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 83
  12. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
38 more
  1. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  2. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 76
  3. 2024Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Imagescited by 69
  4. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  5. 2023Frozen Language Model Helps ECG Zero-Shot Learningcited by 62
  6. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  7. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Modelscited by 61
  8. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  9. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  10. 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understandingcited by 54
  11. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  12. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Textscited by 51
  13. 2024Transcriptomics-guided Slide Representation Learning in Computational Pathologycited by 51
  14. 2024FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Modelscited by 47
  15. 2023Are Natural Domain Foundation Models Useful for Medical Image Classification?cited by 46
  16. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 46
  17. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  18. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  19. 2023CLIP in Medical Imaging: A Surveycited by 40
  20. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  21. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 39
  22. 2024Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammographycited by 38
  23. 2023AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentationcited by 37
  24. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 36
  25. 2024MediCLIP: Adapting CLIP for Few-shot Medical Image Anomaly Detectioncited by 35
  26. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  27. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 33
  28. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  29. 2023Identifying Interpretable Subspaces in Image Representationscited by 32
  30. 2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
  31. 2024Transductive Zero-Shot and Few-Shot CLIPcited by 32
  32. 2023A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosiscited by 31
  33. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  34. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 30
  35. 2024BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrievalcited by 30
  36. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 30
  37. 2024RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reportscited by 30
  38. 2022Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learningcited by 29

50 most cited of 570 All 570 (JSON)