# On the Utility of Speech and Audio Foundation Models for Marmoset Call Analysis

arXiv: https://arxiv.org/abs/2407.16417 · HTML: https://mlsanity.com/paper/2407.16417/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 9 of 9)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation](https://mlsanity.com/paper/2103.06695/) (2021)
- [catch22: CAnonical Time-series CHaracteristics](https://mlsanity.com/paper/1901.10200/) (2019)
- [Computational bioacoustics with deep learning: a review and roadmap](https://mlsanity.com/paper/2112.06725/) (2021)
- [Highly comparative time-series analysis: The empirical structure of time series and their methods](https://mlsanity.com/paper/1304.1209/) (2013)
- [Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers?](https://mlsanity.com/paper/2305.14035/) (2023)

## Built on by (most cited first, 4 of 4)

- [Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing](https://mlsanity.com/paper/2501.05987/) (2025)
- [Crossing the Species Divide: Transfer Learning from Speech to Animal Sounds](https://mlsanity.com/paper/2509.04166/) (2025)
- [Towards Leveraging Sequential Structure in Animal Vocalizations](https://mlsanity.com/paper/2511.10190/) (2025)
- [An Entropy-Guided Curriculum Learning Strategy for Data-Efficient Acoustic Scene Classification under Domain Shift](https://mlsanity.com/paper/2509.11168/) (2025)

## Abstract (arXiv)

Marmoset monkeys encode vital information in their calls and serve as a surrogate model for neuro-biologists to understand the evolutionary origins of human vocal communication. Traditionally analyzed with signal processing-based features, recent approaches have utilized self-supervised models pre-trained on human speech for feature extraction, capitalizing on their ability to learn a signal's intrinsic structure independently of its acoustic domain. However, the utility of such foundation models remains unclear for marmoset call analysis in terms of multi-class classification, bandwidth, and pre-training domain. This study assesses feature representations derived from speech and general audio domains, across pre-training bandwidths of 4, 8, and 16 kHz for marmoset call-type and caller classification tasks. Results show that models with higher bandwidth improve performance, and pre-training on speech or general audio yields comparable results, improving over a spectral baseline.