mlsanityData to 7 Oct 2026

Scaling End-to-End Models for Large-Scale Multilingual ASR

arXiv 2104.14830 ↗First posted 30 Apr 2021cs.CL cs.SD eess.AScited by 61

Not on the map. Why?

Abstract (arXiv)

Building ASR models across many languages is a challenging multi-task learning problem due to large variations and heavily unbalanced data. Existing work has shown positive transfer from high resource to low resource languages. However, degradations on high resource languages are commonly observed due to interference from the heterogeneous multilingual data and reduction in per-language capacity. We conduct a capacity study on a 15-language task, with the amount of data per language varying from 7.6K to 53.5K hours. We adopt GShard [1] to efficiently scale up to 10B parameters. Empirically, we find that (1) scaling the number of model parameters is an effective way to solve the capacity bottleneck - our 500M-param model already outperforms monolingual baselines and scaling it to 1B and 10B brought further quality gains; (2) larger models are not only more data efficient, but also more efficient in terms of training cost as measured in TPU days - the 1B-param model reaches the same accuracy at 34% of training time as the 500M-param model; (3) given a fixed capacity budget, adding depth works better than width and large encoders do better than large decoders; (4) with continuous training, they can be adapted to new languages and domains.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 30 Apr 2021.

Builds on

31 of 43 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  12. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
19 more
  1. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  2. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  3. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  4. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  5. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  6. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  7. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  8. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  9. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  10. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  11. 2020Balancing Training for Multilingual Neural Machine Translationcited by 71
  12. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  13. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  14. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  15. 2020Knowledge Distillation for Multi-task Learningcited by 51
  16. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  17. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
  18. 2019Massively Multilingual Adversarial Speech Recognitioncited by 28
  19. 2018Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languagescited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 12; 2023: 17; 2024: 14; 2025: 4; 2026: 1.

61 papers.

Sort:

By area:

  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  6. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  7. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  8. 2023End-to-End Speech Recognition: A Surveycited by 127
  9. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  10. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  11. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  12. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
38 more
  1. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  2. 2024GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinementcited by 34
  3. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 33
  4. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  5. 2021A Configurable Multilingual Model is All You Need to Recognize All Languagescited by 28
  6. 2022Streaming End-to-End Multilingual Speech Recognition with Joint Language Identificationcited by 24
  7. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  8. 2024Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervisioncited by 17
  9. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16
  10. 2022Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilitiescited by 16
  11. 2024Learn and Don't Forget: Adding a New Language to ASR Foundation Modelscited by 16
  12. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 15
  13. 2022Towards Zero-Shot Code-Switched Speech Recognitioncited by 15
  14. 2024Multi-modal Deep Learningcited by 15
  15. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  16. 2021Enabling On-Device Training of Speech Recognition Models with Federated Dropoutcited by 12
  17. 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
  18. 2024Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Studycited by 12
  19. 2022Improving the fusion of acoustic and text representations in RNN-Tcited by 11
  20. 2022Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translationcited by 11
  21. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  22. 2022A Language Agnostic Multilingual Streaming On-Device ASR Systemcited by 9
  23. 2023Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learningcited by 9
  24. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  25. 2023Unified model for code-switching speech recognition and language identification based on a concatenated tokenizercited by 8
  26. 2022Scaling Up Deliberation for Multilingual ASRcited by 6
  27. 2024Exploring SSL Discrete Tokens for Multilingual ASRcited by 6
  28. 2022Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learningcited by 5
  29. 2024Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Modelscited by 5
  30. 2021A Study of Multilingual End-to-End Speech Recognition for Kazakh, Russian, and Englishcited by 4
  31. 2023SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognitioncited by 4
  32. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 4
  33. 2023UML: A Universal Monolingual Output Layer for Multilingual ASRcited by 3
  34. 2023Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognitioncited by 3
  35. 2023Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encodercited by 3
  36. 2024Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASRcited by 3
  37. 2024A Parameter-efficient Language Extension Framework for Multilingual ASRcited by 3
  38. 2024Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Textcited by 3

50 most cited of 61 All 61 (JSON)