mlsanityData to 7 Oct 2026

Benchmarking Representations for Speech, Music, and Acoustic Events

arXiv 2405.00934 ↗First posted 2 May 2024eess.AS cs.LG cs.SDcited by 14

Not on the map. Why?

Abstract (arXiv)

Limited diversity in standardized benchmarks for evaluating audio representation learning (ARL) methods may hinder systematic comparison of current methods' capabilities. We present ARCH, a comprehensive benchmark for evaluating ARL methods on diverse audio classification domains, covering acoustic events, music, and speech. ARCH comprises 12 datasets, that allow us to thoroughly assess pre-trained SSL models of different sizes. ARCH streamlines benchmarking of ARL techniques through its unified access to a wide range of domains and its ability to readily incorporate new datasets and models. To address the current lack of open-source, pre-trained models for non-speech audio, we also release new pre-trained models that demonstrate strong performance on non-speech datasets. We argue that the presented wide-ranging evaluation provides valuable insights into state-of-the-art ARL methods, and is useful to pinpoint promising research directions.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

26 of 36 references.

References · 26

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  8. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  9. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  10. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  11. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  12. 2016FMA: A Dataset For Music Analysiscited by 205
14 more
  1. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  2. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  3. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  4. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  5. 2018AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmarkcited by 72
  6. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
  7. 2021Towards Learning Universal Audio Representationscited by 51
  8. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  9. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  10. 2022BYOL for Audio: Exploring Pre-trained General-purpose Audio Representationscited by 36
  11. 2016Deep convolutional networks on the pitch spiral for musical instrument recognitioncited by 14
  12. 2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
  13. 2021DECAR: Deep Clustering for learning general-purpose Audio Representationscited by 7
  14. 2022The Efficacy of Self-Supervised Speech Models for Audio Representationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 2.

14 papers.

Sort:

  1. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  2. 2025DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Datasetcited by 8
  3. 2024Voice Disorder Analysis: a Transformer-based Approachcited by 5
  4. 2024Pre-training with Synthetic Patterns for Audiocited by 3
  5. 2025Exploring Generative Error Correction for Dysarthric Speech Recognitioncited by 3
  6. 2025Small is Sufficient: Reducing the World AI Energy Consumption Through Model Selectioncited by 3
  7. 2025MVP: Multi-source Voice Pathology detectioncited by 2
  8. 2026Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddingscited by 2
  9. 2024Contrastive Learning from Synthetic Audio Doppelg"angerscited by 1
  10. 2025Evaluation of Deep Audio Representations for Hearablescited by 1
  11. 2025Optimizing Speech Language Models for Acoustic Consistencycited by 1
  12. 2025AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratchcited by 1
2 more
  1. 2025A Concept-based approach to Voice Disorder Detectioncited by 0
  2. 2026On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluationcited by 0