mlsanityData to 7 Oct 2026

BioCLIP: A Vision Foundation Model for the Tree of Life

arXiv 2311.18803 ↗First posted 30 Nov 2023cs.CV cs.CL cs.LGcited by 159

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Images of the natural world, collected by a variety of cameras, from drones to individual phones, are increasingly abundant sources of biological information. There is an explosion of computational methods and tools, particularly computer vision, for extracting biologically relevant information from images for science and conservation. Yet most of these are bespoke approaches designed for a specific task and are not easily adaptable or extendable to new questions, contexts, and datasets. A vision model for general organismal biology questions on images is of timely need. To approach this, we curate and release TreeOfLife-10M, the largest and most diverse ML-ready dataset of biology images. We then develop BioCLIP, a foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge. We rigorously benchmark our approach on diverse fine-grained biology classification tasks and find that BioCLIP consistently and substantially outperforms existing baselines (by 16% to 17% absolute). Intrinsic evaluation reveals that BioCLIP has learned a hierarchical representation conforming to the tree of life, shedding light on its strong generalizability. https://imageomics.github.io/bioclip has models, data and code.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Builds on 43 of its 95 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

43 of 95 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
31 more
  1. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  2. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  3. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  4. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  5. 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 194
  6. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  7. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  8. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  9. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  10. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  11. 2021Seeing biodiversity: perspectives in machine learning for wildlife conservationcited by 129
  12. 2019Making Better Mistakes: Leveraging Class Hierarchies with Deep Networkscited by 112
  13. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  14. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  15. 2020The iWildCam 2020 Competition Datasetcited by 86
  16. 2021The iWildCam 2021 Competition Datasetcited by 77
  17. 2019PlantDoc: A Dataset for Visual Plant Disease Detectioncited by 65
  18. 2017Do Convolutional Neural Networks Learn Class Hierarchy?cited by 61
  19. 2022Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Frameworkcited by 59
  20. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  21. 2019A deep active learning system for species identification and counting in camera trap imagescited by 44
  22. 2022Contrastive language and vision learning of general fashion conceptscited by 41
  23. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  24. 2023A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-1M Insect Datasetcited by 30
  25. 2023Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classificationcited by 29
  26. 2021Danish Fungi 2020 -- Not Just Another Image Recognition Datasetcited by 27
  27. 2023CiT: Curation in Training for Effective Vision-Language Datacited by 27
  28. 2022On Label Granularity and Object Localizationcited by 12
  29. 2023Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognitioncited by 11
  30. 2023Bird Distribution Modelling using Remote Sensing and Citizen Science datacited by 8
  31. 2019Deep learning on butterfly phenotypes tests evolution's oldest mathematical modelcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 30; 2025: 71; 2026: 57.

159 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2025BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learningcited by 48
  3. 2024TaxaBind: A Unified Embedding Space for Ecological Applicationscited by 33
  4. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  5. 2025Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Priorcited by 28
  6. 2024BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversitycited by 22
  7. 2024Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosiscited by 21
  8. 2024BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversitycited by 19
  9. 2024CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scalecited by 18
  10. 2024Visually Consistent Hierarchical Image Classificationcited by 16
  11. 2025AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmarkcited by 15
  12. 2025Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysiscited by 12
38 more
  1. 2025GARF: Learning Generalizable 3D Reassembly for Real-World Fracturescited by 12
  2. 2024VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Imagescited by 10
  3. 2024Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Imagescited by 9
  4. 2024WildFusion: Individual Animal Identification with Calibrated Similarity Fusioncited by 9
  5. 2024Enhancing Descriptive Captions with Visual Attributes for Multimodal Perceptioncited by 9
  6. 2024Personalized Representation from Personalized Generationcited by 9
  7. 2025CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separationcited by 9
  8. 2024Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolutioncited by 7
  9. 2024FungiTastic: A multi-modal dataset and benchmark for image categorizationcited by 7
  10. 2024Reconstructing Animals and the Wildcited by 7
  11. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  12. 2025TaxaDiffusion: Progressively Trained Diffusion Model for Fine-Grained Species Generationcited by 7
  13. 2024Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Datacited by 6
  14. 2025A Vision-Language Foundation Model for Leaf Disease Identificationcited by 6
  15. 2025The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identificationcited by 6
  16. 2024VLM-CPL: Consensus Pseudo Labels from Vision-Language Models for Annotation-Free Pathological Image Classificationcited by 5
  17. 2024CATALOG: A Camera Trap Language-guided Contrastive Learning Modelcited by 5
  18. 2025AVEX: What Matters for Animal Vocalization Encodingcited by 5
  19. 2023Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphscited by 4
  20. 2024Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decompositioncited by 4
  21. 2024DivShift: Exploring Domain-Specific Distribution Shifts in Large-Scale, Volunteer-Collected Biodiversity Datasetscited by 4
  22. 2025Self-Supervised Pretraining for Fine-Grained Plankton Recognitioncited by 4
  23. 2025Foundation Models for Environmental Science: A Survey of Emerging Frontierscited by 4
  24. 2025GlobalGeoTree: A Multi-Granular Vision-Language Dataset for Global Tree Species Classificationcited by 4
  25. 2025MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reportingcited by 4
  26. 2026Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Studycited by 4
  27. 2026LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for Foundational Vision-Language Understanding of Plant Diseasescited by 4
  28. 2024Towards Zero-Shot Camera Trap Image Categorizationcited by 3
  29. 2024Towards Scalable Insect Monitoring: Ultra-Lightweight CNNs as On-Device Triggers for Insect Camera Trapscited by 3
  30. 2025DiSciPLE: Learning Interpretable Programs for Scientific Visual Discoverycited by 3
  31. 2025A Survey of Foundation Models for Environmental Sciencecited by 3
  32. 2025Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applicationscited by 3
  33. 2025Global and Local Entailment Learning for Natural World Imagerycited by 3
  34. 2025Free-Grained Hierarchical Visual Recognitioncited by 3
  35. 2025BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Modelscited by 3
  36. 2025Modality Alignment across Trees on Heterogeneous Hyperbolic Manifoldscited by 3
  37. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  38. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 3

50 most cited of 159 All 159 (JSON)