mlsanityData to 7 Oct 2026

CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale

arXiv 2405.17537 ↗First posted 27 May 2024cs.AI cs.CL cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse barcode DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 31 of its 81 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

31 of 81 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2020Quantifying Attention Flow in Transformerscited by 759
19 more
  1. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  2. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  3. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  4. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  5. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
  6. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  7. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  8. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  9. 2021Fine-Grained Image Analysis with Deep Learning: A Surveycited by 135
  10. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  11. 2023Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learningcited by 77
  12. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  13. 2020Grafit: Learning fine-grained image representations with coarse labelscited by 43
  14. 2021ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Geneticscited by 35
  15. 2023A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-1M Insect Datasetcited by 30
  16. 2023Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understandingcited by 29
  17. 2021Fine-Grained Zero-Shot Learning with DNA as Side Informationcited by 20
  18. 2022TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrievalcited by 18
  19. 2022SNP2Vec: Scalable Self-Supervised Pre-Training for Genome-Wide Association Studycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversitycited by 22
  2. 2025Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applicationscited by 3
  3. 2025Global and Local Entailment Learning for Natural World Imagerycited by 3
  4. 2025A multi-modal dataset for insect biodiversity with imagery and DNA at the trap and individual levelcited by 3
  5. 2025BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Modelscited by 3
  6. 2025G2PDiffusion: Cross-Species Genotype-to-Phenotype Prediction via Evolutionary Diffusioncited by 2
  7. 2025Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoringcited by 2
  8. 2025Taxonomic Reasoning for Rare Arthropods: Combining Dense Image Captioning and RAG for Interpretable Classificationcited by 1
  9. 2025TerraIncognita: A Dynamic Benchmark for Species Discovery Using Frontier Modelscited by 1
  10. 2025BarcodeMamba+: Advancing State-Space Models for Fungal Biodiversity Researchcited by 1
  11. 2025AquaMonitor: A multimodal multi-view image sequence dataset for real-life aquatic invertebrate biodiversity monitoringcited by 0
  12. 2025Hyperbolic Multimodal Representation Learning for Biological Taxonomiescited by 0
6 more
  1. 2025Bridging Domain Gaps for Fine-Grained Moth Classification Through Expert-Informed Adaptation and Foundation Model Priorscited by 0
  2. 2025What You See is (Usually) What You Get: Multimodal Prototype Networks that Abstain from Expensive Modalitiescited by 0
  3. 2026A continental-scale dataset of ground beetles with high-resolution images and validated morphological trait measurementscited by 0
  4. 2026Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inferencecited by 0
  5. 2026Energy Constrained Hierarchical Underwater Monitoring via Local Multi-Agent RAGcited by 0
  6. 2026InfoTaxa: Information-Calibrated Label-Free Clustering for Fine-Grained Visual Taxonomycited by 0