# AVEX: What Matters for Animal Vocalization Encoding arXiv: https://arxiv.org/abs/2508.11845 ยท HTML: https://mlsanity.com/paper/2508.11845/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 73 references that are in the map. - First posted on arXiv 15 Aug 2025. ## Builds on (most cited first, 30 of 34) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021) - [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021) - [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022) - [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021) - [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022) - [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020) - [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022) - [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023) - [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019) - [Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language](https://mlsanity.com/paper/2212.07525/) (2022) - [Computational bioacoustics with deep learning: a review and roadmap](https://mlsanity.com/paper/2112.06725/) (2021) - [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022) - [AVES: Animal Vocalization Encoder based on Self-Supervision](https://mlsanity.com/paper/2210.14493/) (2022) - [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://mlsanity.com/paper/2401.03497/) (2024) - [Global birdsong embeddings enable superior transfer learning for bioacoustic classification](https://mlsanity.com/paper/2307.06292/) (2023) - [BEANS: The Benchmark of Animal Sounds](https://mlsanity.com/paper/2210.12300/) (2022) - [BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics](https://mlsanity.com/paper/2403.10380/) (2024) - [NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics](https://mlsanity.com/paper/2411.07186/) (2024) - [Can Masked Autoencoders Also Listen to Birds?](https://mlsanity.com/paper/2504.12880/) (2025) - [Transferable Models for Bioacoustics with Human Language Supervision](https://mlsanity.com/paper/2308.04978/) (2023) - [BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics](https://mlsanity.com/paper/2312.07439/) (2023) - [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025) - [animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics](https://mlsanity.com/paper/2406.01253/) (2024) - [Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing](https://mlsanity.com/paper/2501.05987/) (2025) - [IDMT-Traffic: An Open Benchmark Dataset for Acoustic Traffic Monitoring Research](https://mlsanity.com/paper/2104.13620/) (2021) ## Built on by (most cited first, 5 of 5) - [Finding Needles in the Haystack: Transductive Active Labeling in Ecology](https://mlsanity.com/paper/2606.03821/) (2026) - [Beyond the Baseband: Adaptive Multi-Band Encoding for Full-Spectrum Bioacoustics Classification](https://mlsanity.com/paper/2604.27936/) (2026) - [Rhythm of the Deep: Two-tier acoustic organization of sperm-whale codas from click waveforms to second-order sequence dependence](https://mlsanity.com/paper/2606.16084/) (2026) - [Efficient Passive Acoustic Monitoring of Killer Whales Using a Two-Stage Detection and Ecotype Classification Cascade](https://mlsanity.com/paper/2609.01792/) (2026) - [A Strength-Monotonic Law for Domain Alignment in Frozen-Embedding Bioacoustic Classification](https://mlsanity.com/paper/2610.09737/) (2026) ## Abstract (arXiv) Bioacoustics, the study of sounds produced by living organisms, plays a vital role in conservation, biodiversity monitoring, and behavioral studies. Many tasks in this field, such as species, individual, and behavior classification and detection, are well-suited to machine learning. However, they often suffer from limited annotated data, highlighting the need for a general-purpose bioacoustic encoder capable of extracting useful representations for diverse downstream tasks. Such encoders have been proposed before, but are often limited in scope due to a focus on a narrow range of species (typically birds), and a reliance on a single model architecture or training paradigm. Moreover, they are usually evaluated on a small set of tasks and datasets. In this work, we present a large-scale empirical study that covers aspects of bioacoustics that are relevant to research but have previously been scarcely considered: training data diversity and scale, model architectures and training recipes, and the breadth of evaluation tasks and datasets. We obtain encoders that are state-of-the-art on the existing and proposed benchmarks. We also identify what matters for training these encoders, such that this work can be extended when more data are available or better architectures are proposed. Specifically, across 26 datasets with tasks including species classification, detection, individual ID, and vocal repertoire discovery, we find self-supervised pre-training followed by supervised post-training on a mixed bioacoustics + general-audio corpus yields the strongest in- and out-of-distribution performance. We show the importance of data diversity in both stages. To support ongoing research and application, we will release the model checkpoints.