mlsanityData to 7 Oct 2026

BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

arXiv 2303.00915 ↗First posted 2 Mar 2023cs.CV cs.CLcited by 506

Not on the map. Why?

Cited by 182 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Biomedical data is inherently multimodal, comprising physical measurements and natural language narratives. A generalist biomedical AI model needs to simultaneously process different modalities of data, including text and images. Therefore, training an effective generalist biomedical model requires high-quality multimodal data, such as parallel image-text pairs. Here, we present PMC-15M, a novel dataset that is two orders of magnitude larger than existing biomedical multimodal datasets such as MIMIC-CXR, and spans a diverse range of biomedical image types. PMC-15M contains 15 million biomedical image-text pairs collected from 4.4 million scientific articles. Based on PMC-15M, we have pretrained BiomedCLIP, a multimodal foundation model, with domain-specific adaptations tailored to biomedical vision-language processing. We conducted extensive experiments and ablation studies on standard biomedical imaging tasks from retrieval to classification to visual question-answering (VQA). BiomedCLIP achieved new state-of-the-art results in a wide range of standard datasets, substantially outperforming prior approaches. Intriguingly, by large-scale pretraining on diverse biomedical image types, BiomedCLIP even outperforms state-of-the-art radiology-specific models such as BioViL in radiology-specific tasks such as RSNA pneumonia detection. In summary, BiomedCLIP is a fully open-access foundation model that achieves state-of-the-art performance on various biomedical tasks, paving the way for transformative multimodal biomedical discovery and applications. We release our models at https://aka.ms/biomedclip to facilitate future research in multimodal biomedical AI.

Key facts. Cited by 506 papers in the map (data through 7 Oct 2026). Builds on 32 of its 58 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

32 of 58 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
20 more
  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  5. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  7. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  8. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  9. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  10. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  13. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  14. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  15. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  16. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  17. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  18. 2021MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretationcited by 52
  19. 2021Multimodal Representation Learning via Maximization of Local Mutual Informationcited by 27
  20. 2021Self-supervised Image-text Pre-training With Mixed Data In Chest X-rayscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 102; 2025: 188; 2026: 191.

506 papers.

Sort:

By area:

  1. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  2. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  3. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  4. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  5. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  6. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  7. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  8. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 107
  9. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  10. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  11. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  12. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
38 more
  1. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
  2. 2025Vision-Language Models Do Not Understand Negationcited by 69
  3. 2023Generative AI for Medical Imaging: extending the MONAI Frameworkcited by 68
  4. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  5. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  6. 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understandingcited by 53
  7. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  8. 2024FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Modelscited by 47
  9. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 45
  10. 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
  11. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialoguecited by 34
  12. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learningcited by 34
  13. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathologycited by 34
  14. 2025RadGPT: Constructing 3D Image-Text Tumor Datasetscited by 34
  15. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literaturecited by 34
  16. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  17. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  18. 2023BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeyscited by 29
  19. 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 29
  20. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  21. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Modelscited by 28
  22. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaborationcited by 28
  23. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  24. 2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Modelscited by 28
  25. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 28
  26. 2024Towards a Personal Health Large Language Modelcited by 25
  27. 2023Medical Vision Language Pretraining: A surveycited by 24
  28. 2025Pillar-0: A New Frontier for Radiology Foundation Modelscited by 24
  29. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  30. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  31. 2024AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuningcited by 23
  32. 2024BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversitycited by 22
  33. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 22
  34. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  35. 2024MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integrationcited by 21
  36. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  37. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  38. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 19

50 most cited of 506 All 506 (JSON)