# CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale

arXiv: https://arxiv.org/abs/2405.17537 · HTML: https://mlsanity.com/paper/2405.17537/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 81 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [When Does Contrastive Visual Representation Learning Work?](https://mlsanity.com/paper/2105.05837/) (2021)
- [Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning](https://mlsanity.com/paper/2306.01859/) (2023)
- [Towards a Visual-Language Foundation Model for Computational Pathology](https://mlsanity.com/paper/2307.12914/) (2023)
- [Grafit: Learning fine-grained image representations with coarse labels](https://mlsanity.com/paper/2011.12982/) (2020)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-1M Insect Dataset](https://mlsanity.com/paper/2307.10455/) (2023)
- [Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding](https://mlsanity.com/paper/2311.15206/) (2023)
- [Fine-Grained Zero-Shot Learning with DNA as Side Information](https://mlsanity.com/paper/2109.14133/) (2021)
- [TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrieval](https://mlsanity.com/paper/2201.07366/) (2022)

## Built on by (most cited first, 18 of 18)

- [BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity](https://mlsanity.com/paper/2406.12723/) (2024)
- [Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoring](https://mlsanity.com/paper/2503.01691/) (2025)
- [Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applications](https://mlsanity.com/paper/2504.06486/) (2025)
- [Global and Local Entailment Learning for Natural World Imagery](https://mlsanity.com/paper/2506.21476/) (2025)
- [A multi-modal dataset for insect biodiversity with imagery and DNA at the trap and individual level](https://mlsanity.com/paper/2507.06972/) (2025)
- [BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models](https://mlsanity.com/paper/2510.20095/) (2025)
- [G2PDiffusion: Cross-Species Genotype-to-Phenotype Prediction via Evolutionary Diffusion](https://mlsanity.com/paper/2502.04684/) (2025)
- [Taxonomic Reasoning for Rare Arthropods: Combining Dense Image Captioning and RAG for Interpretable Classification](https://mlsanity.com/paper/2503.10886/) (2025)
- [TerraIncognita: A Dynamic Benchmark for Species Discovery Using Frontier Models](https://mlsanity.com/paper/2506.03182/) (2025)
- [BarcodeMamba+: Advancing State-Space Models for Fungal Biodiversity Research](https://mlsanity.com/paper/2512.15931/) (2025)
- [AquaMonitor: A multimodal multi-view image sequence dataset for real-life aquatic invertebrate biodiversity monitoring](https://mlsanity.com/paper/2505.22065/) (2025)
- [Hyperbolic Multimodal Representation Learning for Biological Taxonomies](https://mlsanity.com/paper/2508.16744/) (2025)
- [Bridging Domain Gaps for Fine-Grained Moth Classification Through Expert-Informed Adaptation and Foundation Model Priors](https://mlsanity.com/paper/2508.20089/) (2025)
- [What You See is (Usually) What You Get: Multimodal Prototype Networks that Abstain from Expensive Modalities](https://mlsanity.com/paper/2511.19752/) (2025)
- [A continental-scale dataset of ground beetles with high-resolution images and validated morphological trait measurements](https://mlsanity.com/paper/2601.10687/) (2026)
- [Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference](https://mlsanity.com/paper/2603.25573/) (2026)
- [Energy Constrained Hierarchical Underwater Monitoring via Local Multi-Agent RAG](https://mlsanity.com/paper/2607.24313/) (2026)
- [InfoTaxa: Information-Calibrated Label-Free Clustering for Fine-Grained Visual Taxonomy](https://mlsanity.com/paper/2609.17218/) (2026)

## Abstract (arXiv)

Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse barcode DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.