mlsanityData to 7 Oct 2026

Contrastive Learning of Medical Visual Representations from Paired Images and Text

arXiv 2010.00747 ↗First posted 2 Oct 2020cs.CV cs.CL cs.LGcited by 636

Not on the map. Why?

Cited by 152 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning visual representations of medical images (e.g., X-rays) is core to medical image understanding but its progress has been held back by the scarcity of human annotations. Existing work commonly relies on fine-tuning weights transferred from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. Meanwhile, several recent studies show exciting results from unsupervised contrastive learning from natural images, but we find these methods help little on medical images because of their high inter-class similarity. We propose ConVIRT, an alternative unsupervised strategy to learn medical visual representations by exploiting naturally occurring paired descriptive text. Our new method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test ConVIRT by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that it leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10% as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.

Key facts. Cited by 636 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 85; 2023: 145; 2024: 150; 2025: 117; 2026: 88.

636 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  9. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  10. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  12. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
38 more
  1. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  2. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  3. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  4. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  5. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  7. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  8. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
  9. 2022Post-hoc Concept Bottleneck Modelscited by 289
  10. 2022Prompt Distribution Learningcited by 235
  11. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  12. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  13. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  14. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  15. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  16. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  17. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  18. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  19. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  20. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  21. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
  22. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  23. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  24. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  25. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  26. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  27. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  28. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  29. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  30. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  31. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  32. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  33. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 114
  34. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  35. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  36. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  37. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 100
  38. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96

50 most cited of 636 All 636 (JSON)