# animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics

arXiv: https://arxiv.org/abs/2406.01253 · HTML: https://mlsanity.com/paper/2406.01253/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 100 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Can contrastive learning avoid shortcut solutions?](https://mlsanity.com/paper/2106.11230/) (2021)
- [Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language](https://mlsanity.com/paper/2212.07525/) (2022)

## Built on by (most cited first, 14 of 14)

- [Can Masked Autoencoders Also Listen to Birds?](https://mlsanity.com/paper/2504.12880/) (2025)
- [Foundation Models for Bioacoustics -- a Comparative Review](https://mlsanity.com/paper/2508.01277/) (2025)
- [Clustering and novel class recognition: evaluating bioacoustic deep learning feature extractors](https://mlsanity.com/paper/2504.06710/) (2025)
- [AVEX: What Matters for Animal Vocalization Encoding](https://mlsanity.com/paper/2508.11845/) (2025)
- [Hybrid Disagreement-Diversity Active Learning for Bioacoustic Sound Event Detection](https://mlsanity.com/paper/2505.20956/) (2025)
- [Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings](https://mlsanity.com/paper/2601.08358/) (2026)
- [bacpipe: a Python package to make bioacoustic deep learning models accessible](https://mlsanity.com/paper/2604.11560/) (2026)
- [Explainability of CNN Based Classification Models for Acoustic Signal](https://mlsanity.com/paper/2509.08717/) (2025)
- [From Birdsong to Rumbles: Classifying Elephant Calls with Out-of-Species Embeddings](https://mlsanity.com/paper/2605.00225/) (2026)
- [Greedy Volume Maximization of Gradient Embeddings for Long-Tailed Frame-Level Bioacoustic Active Learning](https://mlsanity.com/paper/2607.13555/) (2026)
- [Efficient Passive Acoustic Monitoring of Killer Whales Using a Two-Stage Detection and Ecotype Classification Cascade](https://mlsanity.com/paper/2609.01792/) (2026)
- [Xeno-Interpretability: Investigating the Alien Minds of LLMs](https://mlsanity.com/paper/2609.20408/) (2026)
- [Active Learning for Biodiversity Monitoring: From Label Efficiency to Reliable Ecological Inference](https://mlsanity.com/paper/2609.27409/) (2026)
- [OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations](https://mlsanity.com/paper/2609.34839/) (2026)

## Abstract (arXiv)

Bioacoustic research, vital for understanding animal behavior, conservation, and ecology, faces a monumental challenge: analyzing vast datasets where animal vocalizations are rare. While deep learning techniques are becoming standard, adapting them to bioacoustics remains difficult. We address this with animal2vec, an interpretable large transformer model, and a self-supervised training scheme tailored for sparse and unbalanced bioacoustic data. It learns from unlabeled audio and then refines its understanding with labeled data. Furthermore, we introduce and publicly release MeerKAT: Meerkat Kalahari Audio Transcripts, a dataset of meerkat (Suricata suricatta) vocalizations with millisecond-resolution annotations, the largest labeled dataset on non-human terrestrial mammals currently available. Our model outperforms existing methods on MeerKAT and the publicly available NIPS4Bplus birdsong dataset. Moreover, animal2vec performs well even with limited labeled data (few-shot learning). animal2vec and MeerKAT provide a new reference point for bioacoustic research, enabling scientists to analyze large amounts of data even with scarce ground truth information.