mlsanityData to 7 Oct 2026

SUPERB: Speech processing Universal PERformance Benchmark

arXiv 2105.01051 ↗First posted 3 May 2021cs.CL cs.SD eess.AScited by 652

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.

Key facts. Cited by 652 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 3 May 2021.

Builds on

32 of 44 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  12. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
20 more
  1. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  2. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  4. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  5. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  6. 2020Unsupervised pretraining transfers well across languagescited by 100
  7. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  8. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  9. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  10. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 86
  11. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  12. 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
  13. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  14. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  15. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  16. 2018wav2letter++: The Fastest Open-source Speech Recognition Systemcited by 42
  17. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  18. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  19. 2019Recent Advances in End-to-End Spoken Language Understandingcited by 13
  20. 2019Problem-Agnostic Speech Embeddings for Multi-Speaker Text-to-Speech with SampleRNNcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 118; 2023: 148; 2024: 137; 2025: 110; 2026: 103.

652 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  3. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  4. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  5. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  6. 2022Masked Autoencoders that Listencited by 271
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  8. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  9. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  10. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  11. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  12. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
38 more
  1. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  2. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  3. 2021Direct speech-to-speech translation with discrete unitscited by 148
  4. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  5. 2022Generative Spoken Dialogue Language Modelingcited by 125
  6. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  7. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  8. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  9. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  10. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  11. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  12. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  13. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  14. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  15. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  16. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  17. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  18. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  19. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  20. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  21. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  22. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  23. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  24. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
  25. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  26. 2023Scaling laws for language encoding models in fMRIcited by 68
  27. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  28. 2022Audio Self-supervised Learning: A Surveycited by 65
  29. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  30. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  31. 2021OpenFL: An open-source framework for Federated Learningcited by 58
  32. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 56
  33. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  34. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 55
  35. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  36. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  37. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 52
  38. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52

50 most cited of 652 All 652 (JSON)