mlsanityData to 7 Oct 2026

PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents

arXiv 2303.07240 ↗First posted 13 Mar 2023cs.CV cs.CL cs.LGcited by 212

Not on the map. Why?

Cited by 94 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Foundation models trained on large-scale dataset gain a recent surge in CV and NLP. In contrast, development in biomedical domain lags far behind due to data scarcity. To address this issue, we build and release PMC-OA, a biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral's OpenAccess subset, which is 8 times larger than before. PMC-OA covers diverse modalities or diseases, with majority of the image-caption samples aligned at finer-grained level, i.e., subfigure and subcaption. While pretraining a CLIP-style model on PMC-OA, our model named PMC-CLIP achieves state-of-the-art results on various downstream tasks, including image-text retrieval on ROCO, MedMNIST image classification, Medical VQA, i.e. +8.1% R@10 on image-text retrieval, +3.9% accuracy on image classification.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

21 of 33 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  12. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
9 more
  1. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  2. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  3. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  4. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  5. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  6. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  7. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  8. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  9. 2021EXSCLAIM! -- An automated pipeline for the construction of labeled materials imaging datasets from literaturecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 57; 2025: 77; 2026: 57.

212 papers.

Sort:

By area:

  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  2. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  3. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  4. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  5. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 194
  6. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  7. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  8. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  9. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 154
  10. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  11. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  12. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
38 more
  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  3. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  4. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  5. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  6. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  7. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  8. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  9. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  10. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcarecited by 62
  11. 2023Large-Vocabulary Segmentation for Medical Images with Text Promptscited by 60
  12. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  13. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathologycited by 57
  14. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  15. 2023PRIOR: Prototype Representation Joint Learning from Medical Images and Reportscited by 55
  16. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  17. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  18. 2024Transcriptomics-guided Slide Representation Learning in Computational Pathologycited by 50
  19. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  20. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 44
  21. 2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
  22. 2023CLIP in Medical Imaging: A Surveycited by 40
  23. 2024MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentationcited by 37
  24. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 37
  25. 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
  26. 2024Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammographycited by 35
  27. 2024WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answeringcited by 34
  28. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literaturecited by 34
  29. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  30. 2024RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reportscited by 29
  31. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaborationcited by 28
  32. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  33. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 28
  34. 2025Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatologycited by 28
  35. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  36. 2023Medical Vision Language Pretraining: A surveycited by 24
  37. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  38. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 23

50 most cited of 212 All 212 (JSON)