mlsanityData to 7 Oct 2026

BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature

arXiv 2501.07171 ↗First posted 13 Jan 2025cs.CV cs.CLcited by 34

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The development of vision-language models (VLMs) is driven by large-scale and diverse multimodal datasets. However, progress toward generalist biomedical VLMs is limited by the lack of annotated, publicly accessible datasets across biology and medicine. Existing efforts are restricted to narrow domains, missing the full diversity of biomedical knowledge encoded in scientific literature. To address this gap, we introduce BIOMEDICA, a scalable, open-source framework to extract, annotate, and serialize the entirety of the PubMed Central Open Access subset into an easy-to-use, publicly accessible dataset. Our framework produces a comprehensive archive with over 24 million unique image-text pairs from over 6 million articles. Metadata and expert-guided annotations are also provided. We demonstrate the utility and accessibility of our resource by releasing BMCA-CLIP, a suite of CLIP-style models continuously pre-trained on the BIOMEDICA dataset via streaming, eliminating the need to download 27 TB of data locally. On average, our models achieve state-of-the-art performance across 40 tasks - spanning pathology, radiology, ophthalmology, dermatology, surgery, molecular biology, parasitology, and cell biology - excelling in zero-shot classification with a 6.56% average improvement (as high as 29.8% and 17.5% in dermatology and ophthalmology, respectively), and stronger image-text retrieval, all while using 10x less compute. To foster reproducibility and collaboration, we release our codebase and dataset for the broader research community.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 70 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

31 of 70 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2021Robust fine-tuning of zero-shot modelscited by 785
19 more
  1. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  2. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  3. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  4. 2023Data Filtering Networkscited by 247
  5. 2024Capabilities of Gemini Models in Medicinecited by 231
  6. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  7. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  8. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 86
  9. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  10. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 68
  11. 2023MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Recordscited by 54
  12. 2024Zero-Shot Clinical Trial Patient Matching with LLMscited by 35
  13. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  14. 2023Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literaturecited by 29
  15. 2018The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionscited by 26
  16. 2024μ-Bench: A Vision-Language Benchmark for Microscopy Understandingcited by 15
  17. 2024Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategiescited by 9
  18. 2024Time-to-Event Pretraining for 3D Medical Imagingcited by 7
  19. 2020Pollen13K: A Large Scale Microscope Pollen Grain Image Datasetcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 20.

34 papers.

Sort:

By area:

  1. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  2. 2025Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learningcited by 9
  3. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  4. 2025Advancing Medical Representation Learning Through High-Quality Datacited by 8
  5. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?cited by 7
  6. 2025Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Modelscited by 5
  7. 2025CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publicationscited by 3
  8. 2025No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Modelscited by 2
  9. 2025Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMscited by 2
  10. 2025HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devicescited by 2
  11. 2026A data- and compute-efficient chest X-ray foundation model beyond aggressive scalingcited by 2
  12. 2025The Impact of Image Resolution on Biomedical Multimodal Large Language Modelscited by 1
22 more
  1. 2025CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selectioncited by 1
  2. 2025Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generationcited by 1
  3. 2025Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Modelscited by 1
  4. 2026Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretrainingcited by 1
  5. 2026MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Modelscited by 1
  6. 2026PhenoLIP: Integrating Phenotype Ontology Knowledge into Medical Vision-Language Pretrainingcited by 0
  7. 2026MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicinecited by 0
  8. 2026LAW & ORDER: Adaptive Spatial Weighting for Medical Diffusion and Segmentationcited by 0
  9. 2026ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Modelscited by 0
  10. 2026LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-raycited by 0
  11. 2026ReXInTheWild: A Unified Benchmark for Medical Photograph Understandingcited by 0
  12. 2026MedP-CLIP: Medical CLIP with Region-Aware Prompt Integrationcited by 0
  13. 2026Scientific Domain Knowledge Improves Vision-Language Fundus Modelscited by 0
  14. 2026Million-scale multimodal pollen microscopy with expert-guided foundation modelscited by 0
  15. 2026MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Modelscited by 0
  16. 2026MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literaturecited by 0
  17. 2026IRIS: An Intelligent Vision-Language System for Ocular Surface Diseases via Topic Tree and Scene-Driven VQA Generationcited by 0
  18. 2026TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Modelscited by 0
  19. 2026A radiographic world model for clinical reasoning and evidence generationcited by 0
  20. 2026Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classificationcited by 0
  21. 2026InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervisioncited by 0
  22. 2026Med-RADIO: Reducing All Medical Domains Into One via Multi-Teacher Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Large-Scale Vision-Language Dataset Derived from Open Scientific Literature to Advance Biomedical Generalist AI
  2. 2024A Refer-and-Ground Multimodal Large Language Model for Biomedicine
  3. 2026MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models
  4. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  5. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models
  6. 2025From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
  7. 2026Aloe-Vision: Robust Vision-Language Models for Healthcare
  8. 2025LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology