mlsanityData to 7 Oct 2026

Towards Robust Speech Representation Learning for Thousands of Languages

arXiv 2407.00837 ↗First posted 30 Jun 2024cs.CL cs.AI cs.SDcited by 48

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. However, models are still far from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual Encoder for Universal Speech, trained on over 1 million hours of data across 4057 languages, extending the language coverage of SSL models 4-fold. We combine 1 million hours of speech from existing publicly accessible corpora with a newly created corpus of 7400+ hours from 4057 languages, which will be publicly released. To handle the diverse conditions of multilingual speech data, we augment the typical SSL masked prediction approach with a novel dereverberation objective, increasing robustness. We evaluate XEUS on several benchmarks, and show that it consistently outperforms or achieves comparable results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or pre-training data. Checkpoints, code, and data are found in https://www.wavlab.org/activities/2024/xeus/.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 62 of its 90 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

62 of 90 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
38 more
  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  6. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  7. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  11. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  12. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  13. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  14. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  15. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  16. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  17. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  18. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  19. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  20. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  21. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
  22. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  23. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  24. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  25. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  26. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  27. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  28. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  29. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  30. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  31. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 56
  32. 2021Interspeech 2021 Deep Noise Suppression Challengecited by 43
  33. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  34. 2022Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Datasetcited by 36
  35. 2023The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASRcited by 36
  36. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
  37. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  38. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 28; 2026: 17.

48 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2025ZIPA: A family of efficient models for multilingual phone recognitioncited by 18
  3. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 17
  4. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  5. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  6. 2024AASIST3: KAN-Enhanced AASIST Speech Deepfake Detection using SSL Features and Additional Regularization for the ASVspoof 2024 Challengecited by 8
  7. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  8. 2024SpeechTaxi: On Multilingual Semantic Speech Classificationcited by 6
  9. 2025Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnetcited by 6
  10. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  11. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systemscited by 5
  12. 2025A Practitioner's Guide to Building ASR Models for Low-Resource Languages: A Case Study on Scottish Gaeliccited by 5
36 more
  1. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  2. 2026An Empirical Recipe for Universal Phone Recognitioncited by 5
  3. 2025Crossing the Species Divide: Transfer Learning from Speech to Animal Soundscited by 4
  4. 2025The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varietiescited by 4
  5. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 4
  6. 2025KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularizationcited by 3
  7. 2025Double Entendre: Robust Audio-Based AI-Generated Lyrics Detection via Multi-View Fusioncited by 3
  8. 2025SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancementcited by 3
  9. 2025EZ-VC: Easy Zero-shot Any-to-Any Voice Conversioncited by 2
  10. 2025In-context Language Learning for Endangered Languages in Speech Recognitioncited by 2
  11. 2025CAM~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguesecited by 2
  12. 2025Learning Robust Spatial Representations from Binaural Audio through Feature Distillationcited by 2
  13. 2025Towards Unsupervised Speech Recognition at the Syllable-Levelcited by 2
  14. 2025CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognitioncited by 1
  15. 2025Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restorationcited by 1
  16. 2025Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTCcited by 1
  17. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  18. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 1
  19. 2026Kakugo: Distillation of Low-Resource Languages into Small Language Modelscited by 1
  20. 2026BranchShine: Compact Raw-Audio-to-IPA Transcription with a RoPE E-Branchformer Encodercited by 1
  21. 2025From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognitioncited by 0
  22. 2025Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025cited by 0
  23. 2025Geolocation-Aware Robust Spoken Language Identificationcited by 0
  24. 2025An Entropy-Guided Curriculum Learning Strategy for Data-Efficient Acoustic Scene Classification under Domain Shiftcited by 0
  25. 2026DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Unitscited by 0
  26. 2026Saar-Voice: A Multi-Speaker Saarbr"ucken Dialect Speech Corpuscited by 0
  27. 2026FalAR: A Large-scale Speaker-Annotated European Portuguese Speech Corpus of Parliamentary Sessionscited by 0
  28. 2026USAD 2.0: Scaling Representation Distillation for Universal Audio Understandingcited by 0
  29. 2026UR-BERT: Scaling Text Encoders for Massively Multilingual TTS Through Universal Romanization and Speech Token Predictioncited by 0
  30. 2026VeRe-Flow: Guiding Flow Matching toward Clean Speech via Velocity Contrastive Regularization and Representation Alignment for Noise-Robust Bandwidth Expansioncited by 0
  31. 2026An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generationcited by 0
  32. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0
  33. 2026BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Modelscited by 0
  34. 2026TokenMapper: A Step Toward Interoperable Speech Token Translationcited by 0
  35. 2026BranchShine-CR: Compact Multilingual IPA Transcription with Self-Conditioned CTC and Consistency Regularizationcited by 0
  36. 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0