mlsanityData to 14 Oct 2026

The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties

arXiv 2509.07139 ↗First posted 8 Sep 2025cs.CL eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Recent improvements in multilingual ASR have not been equally distributed across languages and language varieties. To advance state-of-the-art (SOTA) ASR models, we present the Interspeech 2025 ML-SUPERB 2.0 Challenge. We construct a new test suite that consists of data from 200+ languages, accents, and dialects to evaluate SOTA multilingual speech models. The challenge also introduces an online evaluation server based on DynaBench, allowing for flexibility in model design and architecture for participants. The challenge received 5 submissions from 3 teams, all of which outperformed our baselines. The best-performing submission achieved an absolute improvement in LID accuracy of 23% and a reduction in CER of 18% when compared to the best baseline on a general multilingual test set. On accented and dialectal data, the best submission obtained 30.2% lower CER and 15.7% higher LID accuracy, showing the importance of community challenges in making speech technologies more inclusive.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

35 of 49 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
23 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  2. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  4. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  5. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  6. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  7. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  8. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  9. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 50
  10. 2024GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speechcited by 30
  11. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 27
  12. 2021SwissDial: Parallel Multidialectal Corpus of Spoken Swiss Germancited by 26
  13. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  14. 2024Towards measuring fairness in speech recognition: Fair-Speech datasetcited by 26
  15. 2024Casablanca: Data and Models for Multidialectal Arabic Speech Recognitioncited by 20
  16. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  17. 2025OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Modelscited by 17
  18. 2022Automatic Speech Recognition Datasets in Cantonese: A Survey and New Datasetcited by 12
  19. 2024The Interspeech 2024 Challenge on Speech Processing Using Discrete Unitscited by 12
  20. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  21. 2021Finnish Dialect Identification: The Effect of Audio and Textcited by 8
  22. 2023Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognitioncited by 7
  23. 2024Self-supervised Speech Representations Still Struggle with African American Vernacular Englishcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 2
  2. 2025How I Built ASR for Endangered Languages with a Spoken Dictionarycited by 1
  3. 2025Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuningcited by 0
  4. 2026GigaAM Multilingual: Foundation Model for Underrepresented Languagescited by 0