mlsanityData to 7 Oct 2026

Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities

arXiv 2211.05756 ↗First posted 10 Nov 2022cs.CL cs.SD eess.AScited by 16

Not on the map. Why?

Abstract (arXiv)

End-to-end multilingual ASR has become more appealing because of several reasons such as simplifying the training and deployment process and positive performance transfer from high-resource to low-resource languages. However, scaling up the number of languages, total hours, and number of unique tokens is not a trivial task. This paper explores large-scale multilingual ASR models on 70 languages. We inspect two architectures: (1) Shared embedding and output and (2) Multiple embedding and output model. In the shared model experiments, we show the importance of tokenization strategy across different languages. Later, we use our optimal tokenization strategy to train multiple embedding and output model to further improve our result. Our multilingual ASR achieves 13.9%-15.6% average WER relative improvement compared to monolingual models. We show that our multilingual ASR generalizes well on an unseen dataset and domain, achieving 9.5% and 7.5% WER on Multilingual Librispeech (MLS) with zero-shot and finetuning, respectively.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

29 of 36 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Layer Normalizationcited by 5,579
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  8. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  11. 2015Listen, Attend and Spellcited by 542
  12. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
17 more
  1. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  2. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  3. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  4. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  5. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  6. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  7. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  8. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 95
  9. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  10. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  11. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  12. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
  13. 2018Improved training of end-to-end attention models for speech recognitioncited by 35
  14. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 15
  15. 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
  16. 2018A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinesecited by 10
  17. 2019Exploring Lexicon-Free Modeling Units for End-to-End Korean and Korean-English Code-Switching Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 7; 2026: 1.

16 papers.

Sort:

  1. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  2. 2024IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languagescited by 39
  3. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  4. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  5. 2024Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervisioncited by 17
  6. 2023Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Modelcited by 3
  7. 2024Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Promptingcited by 3
  8. 2025LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialectcited by 3
  9. 2025LLM-based phoneme-to-grapheme for phoneme-based speech recognitioncited by 3
  10. 2025Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximationcited by 3
  11. 2025Swivuriso: The South African Next Voices Multilingual Speech Datasetcited by 3
  12. 2025Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levelscited by 2
4 more
  1. 2024Romanization Encoding For Multilingual ASRcited by 1
  2. 2025ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognitioncited by 1
  3. 2025The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASRcited by 0
  4. 2026Advancing LLM-based phoneme-to-grapheme for multilingual speech recognitioncited by 0