# Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers?

arXiv: https://arxiv.org/abs/2305.14035 · HTML: https://mlsanity.com/paper/2305.14035/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 18 of 18)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Uncovering the structure of clinical EEG signals with self-supervised learning](https://mlsanity.com/paper/2007.16104/) (2020)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [Computational bioacoustics with deep learning: a review and roadmap](https://mlsanity.com/paper/2112.06725/) (2021)
- [Vector-Quantized Autoregressive Predictive Coding](https://mlsanity.com/paper/2005.08392/) (2020)
- [Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies](https://mlsanity.com/paper/2011.00406/) (2020)
- [Self-supervised representation learning from electroencephalography signals](https://mlsanity.com/paper/1911.05419/) (2019)
- [Multi-Task Self-Supervised Pre-Training for Music Classification](https://mlsanity.com/paper/2102.03229/) (2021)

## Built on by (most cited first, 9 of 9)

- [Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing](https://mlsanity.com/paper/2501.05987/) (2025)
- [On the Utility of Speech and Audio Foundation Models for Marmoset Call Analysis](https://mlsanity.com/paper/2407.16417/) (2024)
- [Feature Representations for Automatic Meerkat Vocalization Classification](https://mlsanity.com/paper/2408.15296/) (2024)
- [Synthetic data enables context-aware bioacoustic sound event detection](https://mlsanity.com/paper/2503.00296/) (2025)
- [No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings](https://mlsanity.com/paper/2508.10230/) (2025)
- [Towards Leveraging Sequential Structure in Animal Vocalizations](https://mlsanity.com/paper/2511.10190/) (2025)
- [Learning Marmoset Vocal Patterns with a Masked Autoencoder for Robust Call Segmentation, Classification, and Caller Identification](https://mlsanity.com/paper/2410.23279/) (2024)
- [GmSLM : Generative Marmoset Spoken Language Modeling](https://mlsanity.com/paper/2509.09198/) (2025)
- [Beyond the Baseband: Adaptive Multi-Band Encoding for Full-Spectrum Bioacoustics Classification](https://mlsanity.com/paper/2604.27936/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) models use only the intrinsic structure of a given signal, independent of its acoustic domain, to extract essential information from the input to an embedding space. This implies that the utility of such representations is not limited to modeling human speech alone. Building on this understanding, this paper explores the cross-transferability of SSL neural representations learned from human speech to analyze bio-acoustic signals. We conduct a caller discrimination analysis and a caller detection study on Marmoset vocalizations using eleven SSL models pre-trained with various pretext tasks. The results show that the embedding spaces carry meaningful caller information and can successfully distinguish the individual identities of Marmoset callers without fine-tuning. This demonstrates that representations pre-trained on human speech can be effectively applied to the bio-acoustics domain, providing valuable insights for future investigations in this field.