# Benchmarking Representations for Speech, Music, and Acoustic Events

arXiv: https://arxiv.org/abs/2405.00934 · HTML: https://mlsanity.com/paper/2405.00934/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 26 of 26)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmark](https://mlsanity.com/paper/1807.03418/) (2018)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning](https://mlsanity.com/paper/2101.10803/) (2021)
- [LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech](https://mlsanity.com/paper/2104.11462/) (2021)
- [BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations](https://mlsanity.com/paper/2204.07402/) (2022)
- [Deep convolutional networks on the pitch spiral for musical instrument recognition](https://mlsanity.com/paper/1605.06644/) (2016)
- [Pretext Tasks selection for multitask self-supervised speech representation learning](https://mlsanity.com/paper/2107.00594/) (2021)
- [DECAR: Deep Clustering for learning general-purpose Audio Representations](https://mlsanity.com/paper/2110.08895/) (2021)
- [The Efficacy of Self-Supervised Speech Models for Audio Representations](https://mlsanity.com/paper/2209.12900/) (2022)

## Built on by (most cited first, 15 of 15)

- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset](https://mlsanity.com/paper/2505.19978/) (2025)
- [Voice Disorder Analysis: a Transformer-based Approach](https://mlsanity.com/paper/2406.14693/) (2024)
- [Pre-training with Synthetic Patterns for Audio](https://mlsanity.com/paper/2410.00511/) (2024)
- [Exploring Generative Error Correction for Dysarthric Speech Recognition](https://mlsanity.com/paper/2505.20163/) (2025)
- [Small is Sufficient: Reducing the World AI Energy Consumption Through Model Selection](https://mlsanity.com/paper/2510.01889/) (2025)
- [MVP: Multi-source Voice Pathology detection](https://mlsanity.com/paper/2505.20050/) (2025)
- [Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings](https://mlsanity.com/paper/2601.08358/) (2026)
- [Contrastive Learning from Synthetic Audio Doppelg"angers](https://mlsanity.com/paper/2406.05923/) (2024)
- [Evaluation of Deep Audio Representations for Hearables](https://mlsanity.com/paper/2502.06664/) (2025)
- [Optimizing Speech Language Models for Acoustic Consistency](https://mlsanity.com/paper/2509.26276/) (2025)
- [AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch](https://mlsanity.com/paper/2510.19368/) (2025)
- [On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation](https://mlsanity.com/paper/2601.06329/) (2026)
- [A Concept-based approach to Voice Disorder Detection](https://mlsanity.com/paper/2507.17799/) (2025)
- [Multimodal Co-Generation With Continuous Bitstream Diffusion](https://mlsanity.com/paper/2610.14349/) (2026)

## Abstract (arXiv)

Limited diversity in standardized benchmarks for evaluating audio representation learning (ARL) methods may hinder systematic comparison of current methods' capabilities. We present ARCH, a comprehensive benchmark for evaluating ARL methods on diverse audio classification domains, covering acoustic events, music, and speech. ARCH comprises 12 datasets, that allow us to thoroughly assess pre-trained SSL models of different sizes. ARCH streamlines benchmarking of ARL techniques through its unified access to a wide range of domains and its ability to readily incorporate new datasets and models. To address the current lack of open-source, pre-trained models for non-speech audio, we also release new pre-trained models that demonstrate strong performance on non-speech datasets. We argue that the presented wide-ranging evaluation provides valuable insights into state-of-the-art ARL methods, and is useful to pinpoint promising research directions.