mlsanityData to 14 Oct 2026

BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models

arXiv 2510.20095 ↗First posted 23 Oct 2025cs.CV cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary samples from the latent morphospace of a species, each capturing certain biological traits. Incorporating captions during training encourages alignment with this shared latent structure, emphasizing potentially diagnostic characters while suppressing spurious correlations. The main challenge, however, lies in obtaining faithful, instance-specific captions at scale. This requirement has limited the utilization of natural language supervision in organismal biology compared with many other scientific domains. We complement this gap by generating synthetic captions with multimodal large language models (MLLMs), guided by Wikipedia-derived visual information and taxon-tailored format examples. These domain-specific contexts help reduce hallucination and yield accurate, instance-based descriptive captions. Using these captions, we train BioCAP (i.e., BioCLIP with Captions), a biological foundation model that captures rich semantics and achieves strong performance in species classification and text-image retrieval. These results demonstrate the value of descriptive captions beyond labels in bridging biological images with multimodal foundation models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 45 of its 75 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

45 of 75 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
33 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  4. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  5. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  7. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  8. 2023Improving CLIP Training with Language Rewritescited by 217
  9. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  10. 2023GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localizationcited by 187
  11. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  12. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  13. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 161
  14. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  15. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  16. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  17. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Areacited by 71
  18. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  19. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70
  20. 2019PlantDoc: A Dataset for Visual Plant Disease Detectioncited by 68
  21. 2025BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learningcited by 50
  22. 2024TaxaBind: A Unified Embedding Space for Ecological Applicationscited by 35
  23. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  24. 2023Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classificationcited by 30
  25. 2023A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-1M Insect Datasetcited by 30
  26. 2021Danish Fungi 2020 -- Not Just Another Image Recognition Datasetcited by 27
  27. 2024BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversitycited by 24
  28. 2024BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversitycited by 20
  29. 2024CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scalecited by 18
  30. 2023Hyp-OW: Exploiting Hierarchical Structure Learning with Hyperbolic Distance Enhances Open World Object Detectioncited by 15
  31. 2025Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanationcited by 15
  32. 2022Choosing an Appropriate Platform and Workflow for Processing Camera Trap Data using Artificial Intelligencecited by 6
  33. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
  2. 2025Visual Species Recognition with Large Multimodal Models as Post-Hoc Correctorscited by 0
  3. 2026Beyond Flat Labels: Level-Restricted Contrastive Learning for Hierarchical Fine-Grained Vision Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining
  2. 2026TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life
  3. 2024VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images
  4. 2025No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models
  5. 2025FishNet++: Analyzing the capabilities of Multimodal Large Language Models in marine biology
  6. 2026MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph
  7. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature
  8. 2024Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models