# AVES: Animal Vocalization Encoder based on Self-Supervision

arXiv: https://arxiv.org/abs/2210.14493 · HTML: https://mlsanity.com/paper/2210.14493/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2022.

## Built on by (most cited first, 30 of 53)

- [Can Masked Autoencoders Also Listen to Birds?](https://mlsanity.com/paper/2504.12880/) (2025)
- [Perch 2.0: The Bittern Lesson for Bioacoustics](https://mlsanity.com/paper/2508.04665/) (2025)
- [Transferable Models for Bioacoustics with Human Language Supervision](https://mlsanity.com/paper/2308.04978/) (2023)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics](https://mlsanity.com/paper/2406.01253/) (2024)
- [Foundation Models for Bioacoustics -- a Comparative Review](https://mlsanity.com/paper/2508.01277/) (2025)
- [AnuraSet: A dataset for benchmarking Neotropical anuran calls identification in passive acoustic monitoring](https://mlsanity.com/paper/2307.06860/) (2023)
- [Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing](https://mlsanity.com/paper/2501.05987/) (2025)
- [Clustering and novel class recognition: evaluating bioacoustic deep learning feature extractors](https://mlsanity.com/paper/2504.06710/) (2025)
- [Perch 2.0 transfers 'whale' to underwater tasks](https://mlsanity.com/paper/2512.03219/) (2025)
- [Exploring bat song syllable representations in self-supervised audio encoders](https://mlsanity.com/paper/2409.12634/) (2024)
- [AVEX: What Matters for Animal Vocalization Encoding](https://mlsanity.com/paper/2508.11845/) (2025)
- [ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Sounds](https://mlsanity.com/paper/2402.03269/) (2024)
- [Robust detection of overlapping bioacoustic sound events](https://mlsanity.com/paper/2503.02389/) (2025)
- [Crossing the Species Divide: Transfer Learning from Speech to Animal Sounds](https://mlsanity.com/paper/2509.04166/) (2025)
- [WhaleNet: a Novel Deep Learning Architecture for Marine Mammals Vocalizations on Watkins Marine Mammal Sound Database](https://mlsanity.com/paper/2402.17775/) (2024)
- [Acoustic identification of individual animals with hierarchical contrastive learning](https://mlsanity.com/paper/2409.08673/) (2024)
- [FinchGPT: a Transformer based language model for birdsong analysis](https://mlsanity.com/paper/2502.00344/) (2025)
- [Synthetic data enables context-aware bioacoustic sound event detection](https://mlsanity.com/paper/2503.00296/) (2025)
- [Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Models](https://mlsanity.com/paper/2511.05171/) (2025)
- [Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings](https://mlsanity.com/paper/2601.08358/) (2026)
- [bacpipe: a Python package to make bioacoustic deep learning models accessible](https://mlsanity.com/paper/2604.11560/) (2026)
- [Dolph2Vec: Self-Supervised Representations of Dolphin Vocalizations](https://mlsanity.com/paper/2606.12503/) (2026)
- [Biospheric AI](https://mlsanity.com/paper/2401.17805/) (2024)
- [Phonetic and Lexical Discovery of a Canine Language using HuBERT](https://mlsanity.com/paper/2402.15985/) (2024)
- [No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings](https://mlsanity.com/paper/2508.10230/) (2025)
- [Data-Efficient Self-Supervised Algorithms for Fine-Grained Birdsong Analysis](https://mlsanity.com/paper/2511.12158/) (2025)
- [WhAM: Towards A Translative Model of Sperm Whale Vocalization](https://mlsanity.com/paper/2512.02206/) (2025)
- [From Birdsong to Rumbles: Classifying Elephant Calls with Out-of-Species Embeddings](https://mlsanity.com/paper/2605.00225/) (2026)
- [Multi-layer attentive probing improves transfer of audio representations for bioacoustics](https://mlsanity.com/paper/2605.10494/) (2026)

## Abstract (arXiv)

The lack of annotated training data in bioacoustics hinders the use of large-scale neural network models trained in a supervised way. In order to leverage a large amount of unannotated audio data, we propose AVES (Animal Vocalization Encoder based on Self-Supervision), a self-supervised, transformer-based audio representation model for encoding animal vocalizations. We pretrain AVES on a diverse set of unannotated audio datasets and fine-tune them for downstream bioacoustics tasks. Comprehensive experiments with a suite of classification and detection tasks have shown that AVES outperforms all the strong baselines and even the supervised "topline" models trained on annotated audio classification datasets. The results also suggest that curating a small training subset related to downstream tasks is an efficient way to train high-quality audio representation models. We open-source our models at urlhttps://github.com/earthspecies/aves.