mlsanityData to 7 Oct 2026

SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities

arXiv 2203.06849 ↗First posted 14 Mar 2022cs.CL cs.SD eess.AScited by 76

Not on the map. Why?

Abstract (arXiv)

Transfer learning has proven to be crucial in advancing the state of speech and natural language processing research in recent years. In speech, a model pre-trained by self-supervised learning transfers remarkably well on multiple tasks. However, the lack of a consistent evaluation methodology is limiting towards a holistic understanding of the efficacy of such models. SUPERB was a step towards introducing a common benchmark to evaluate pre-trained models across various speech tasks. In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain and quality across different types of tasks. It entails freezing pre-trained model parameters, only using simple task-specific trainable heads. The goal is to be inclusive of all researchers, and encourage efficient use of computational resources. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 49 of its 59 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

49 of 59 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
37 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  12. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  13. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  14. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  15. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  16. 2016Exploring the Limits of Language Modelingcited by 172
  17. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  18. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  19. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  20. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  21. 2020Unsupervised pretraining transfers well across languagescited by 100
  22. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  23. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  24. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  25. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  26. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  27. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  28. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  29. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  30. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  31. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  32. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  33. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  34. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 22
  35. 2020On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognitioncited by 19
  36. 2020Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representationscited by 18
  37. 2019Problem-Agnostic Speech Embeddings for Multi-Speaker Text-to-Speech with SampleRNNcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 23; 2024: 17; 2025: 14; 2026: 6.

76 papers.

Sort:

By area:

  1. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  2. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  3. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  4. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  5. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  6. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  7. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  8. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  9. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 38
  10. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  11. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  12. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
38 more
  1. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  2. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  3. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  4. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  5. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  6. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  7. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  8. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  9. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  10. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  11. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
  12. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  13. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  14. 2022Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translationcited by 11
  15. 2022A Comparative Study of Self-supervised Speech Representation Based Voice Conversioncited by 9
  16. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
  17. 2024On the Evaluation of Speech Foundation Models for Spoken Language Understandingcited by 9
  18. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  19. 2023MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Modelscited by 8
  20. 2024UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalizationcited by 8
  21. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 8
  22. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  23. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7
  24. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  25. 2023Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speechcited by 6
  26. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  27. 2022Speech separation with large-scale self-supervised learningcited by 5
  28. 2024Interface Design for Self-Supervised Speech Modelscited by 5
  29. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  30. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  31. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  32. 2022T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5cited by 4
  33. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  34. 2024MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyondcited by 4
  35. 2025The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varietiescited by 4
  36. 2022Bridging Speech and Textual Pre-trained Models with Unsupervised ASRcited by 3
  37. 2022Introducing Semantics into Speech Encoderscited by 3
  38. 2023Large-scale unsupervised audio pre-training for video-to-speech synthesiscited by 3

50 most cited of 76 All 76 (JSON)