mlsanityData to 7 Oct 2026

Scaling Speech Technology to 1,000+ Languages

arXiv 2305.13516 ↗First posted 22 May 2023cs.CL cs.SD eess.AScited by 401

Not on the map. Why?

Abstract (arXiv)

Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages. Experiments show that our multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

Key facts. Cited by 401 papers in the map (data through 7 Oct 2026). Builds on 70 of its 106 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

70 of 106 references.

References · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2016Layer Normalizationcited by 5,579
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2016Density estimation using Real NVPcited by 2,029
38 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  13. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  14. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  15. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  16. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  17. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  18. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  19. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  20. 2021A Survey on Neural Speech Synthesiscited by 182
  21. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  22. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  23. 2021Unsupervised Speech Recognitioncited by 147
  24. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  25. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  26. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  27. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  28. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  29. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  30. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  31. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  32. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  33. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 75
  34. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  35. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  36. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  37. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  38. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 110; 2025: 127; 2026: 128.

401 papers.

Sort:

By area:

  1. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  2. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  3. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  4. 2023A Survey on Detection of LLMs-Generated Contentcited by 70
  5. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  6. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  7. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  8. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 56
  9. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  10. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  11. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  12. 2024IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languagescited by 39
38 more
  1. 2023Can Whisper perform speech-based in-context learning?cited by 35
  2. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  3. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
  4. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  5. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  6. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  7. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  8. 2024Multilingual Turn-taking Prediction Using Voice Activity Projectioncited by 26
  9. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  10. 2023Turning Whisper into Real-Time Transcription Systemcited by 23
  11. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  12. 2025LongCat-Flash-Omni Technical Reportcited by 23
  13. 2024Advocating Character Error Rate for Multilingual ASR Evaluationcited by 22
  14. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 22
  15. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 20
  16. 2024Casablanca: Data and Models for Multidialectal Arabic Speech Recognitioncited by 20
  17. 2024HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Modelscited by 19
  18. 2024A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatarscited by 18
  19. 2025ZIPA: A family of efficient models for multilingual phone recognitioncited by 18
  20. 2024Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervisioncited by 17
  21. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 17
  22. 2023Self-Supervised Models of Speech Infer Universal Articulatory Kinematicscited by 16
  23. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  24. 2024YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detectioncited by 16
  25. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  26. 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Expertscited by 15
  27. 2024Towards Hierarchical Spoken Language Dysfluency Modelingcited by 15
  28. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  29. 2023ArTST: Arabic Text and Speech Transformercited by 14
  30. 2023The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any languagecited by 14
  31. 2024Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacycited by 14
  32. 2025NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inferencecited by 14
  33. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 14
  34. 2025POWSM: A Phonetic Open Whisper-Style Speech Foundation Modelcited by 14
  35. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  36. 2024PhoWhisper: Automatic Speech Recognition for Vietnamesecited by 13
  37. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  38. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13

50 most cited of 401 All 401 (JSON)