# BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

arXiv: https://arxiv.org/abs/2603.23883 · HTML: https://mlsanity.com/paper/2603.23883/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [AP-10K: A Benchmark for Animal Pose Estimation in the Wild](https://mlsanity.com/paper/2108.12617/) (2021)
- [Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding](https://mlsanity.com/paper/2204.08129/) (2022)
- [Computational bioacoustics with deep learning: a review and roadmap](https://mlsanity.com/paper/2112.06725/) (2021)
- [Cross-X Learning for Fine-Grained Visual Categorization](https://mlsanity.com/paper/1909.04412/) (2019)
- [Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition](https://mlsanity.com/paper/1806.05372/) (2018)
- [BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning](https://mlsanity.com/paper/2505.23883/) (2025)
- [WildlifeDatasets: An open-source toolkit for animal re-identification](https://mlsanity.com/paper/2311.09118/) (2023)
- [BEANS: The Benchmark of Animal Sounds](https://mlsanity.com/paper/2210.12300/) (2022)
- [TaxaBind: A Unified Embedding Space for Ecological Applications](https://mlsanity.com/paper/2411.00683/) (2024)
- [INQUIRE: A Natural World Text-to-Image Retrieval Benchmark](https://mlsanity.com/paper/2411.02537/) (2024)
- [NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics](https://mlsanity.com/paper/2411.07186/) (2024)
- [SeaTurtleID2022: A long-span dataset for reliable sea turtle re-identification](https://mlsanity.com/paper/2211.10307/) (2022)
- [Transferable Models for Bioacoustics with Human Language Supervision](https://mlsanity.com/paper/2308.04978/) (2023)
- [Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images](https://mlsanity.com/paper/2307.15904/) (2023)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [PetFace: A Large-Scale Dataset and Benchmark for Animal Identification](https://mlsanity.com/paper/2407.13555/) (2024)
- [Exploring Fine-Grained Audiovisual Categorization with the SSW60 Dataset](https://mlsanity.com/paper/2207.10664/) (2022)
- [GEOBIND: Binding Text, Image, and Audio through Satellite Images](https://mlsanity.com/paper/2404.11720/) (2024)
- [OpenAnimalTracks: A Dataset for Animal Track Recognition](https://mlsanity.com/paper/2406.09647/) (2024)
- [MAviS: A Multimodal Conversational Assistant For Avian Species](https://mlsanity.com/paper/2603.07294/) (2026)
- [AnimalClue: Recognizing Animals by their Traces](https://mlsanity.com/paper/2507.20240/) (2025)
- [AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference](https://mlsanity.com/paper/2603.22053/) (2026)

## Abstract (arXiv)

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/