mlsanityData to 7 Oct 2026

A Configurable Multilingual Model is All You Need to Recognize All Languages

arXiv 2107.05876 ↗First posted 13 Jul 2021eess.AS cs.CL cs.SDcited by 28

Not on the map. Why?

Abstract (arXiv)

Multilingual automatic speech recognition (ASR) models have shown great promise in recent years because of the simplified model training and deployment process. Conventional methods either train a universal multilingual model without taking any language information or with a 1-hot language ID (LID) vector to guide the recognition of the target language. In practice, the user can be prompted to pre-select several languages he/she can speak. The multilingual model without LID cannot well utilize the language information set by the user while the multilingual model with LID can only handle one pre-selected language. In this paper, we propose a novel configurable multilingual model (CMM) which is trained only once but can be configured as different models based on users' choices by extracting language-specific modules together with a universal model from the trained CMM. Particularly, a single CMM can be deployed to any user scenario where the users can pre-select any combination of languages. Trained with 75K hours of transcribed anonymized Microsoft multilingual data and evaluated with 10-language test sets, the proposed CMM improves from the universal multilingual model by 26.0%, 16.9%, and 10.4% relative word error reduction when the user selects 1, 2, or 3 languages, respectively. CMM also performs significantly better on code-switching test sets.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

19 of 33 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Listen, Attend and Spellcited by 542
  3. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  4. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  5. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  6. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  7. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  8. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  9. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  10. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  11. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  12. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
7 more
  1. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  2. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
  3. 2020Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognitioncited by 28
  4. 2020Transformer-Transducers for Code-Switched Speech Recognitioncited by 27
  5. 2020Meta-Transfer Learning for Code-Switched Speech Recognitioncited by 22
  6. 2020Benchmarking LF-MMI, CTC and RNN-T Criteria for Streaming ASRcited by 11
  7. 2018Developing Far-Field Speaker System Via Teacher-Student Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 8; 2024: 6; 2025: 2; 2026: 1.

28 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
  3. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  4. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  5. 2022Streaming End-to-End Multilingual Speech Recognition with Joint Language Identificationcited by 24
  6. 2022LAE: Language-Aware Encoder for Monolingual and Multilingual ASRcited by 21
  7. 2021Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorizationcited by 17
  8. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  9. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16
  10. 2022Towards Zero-Shot Code-Switched Speech Recognitioncited by 15
  11. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12
  12. 2022A Language Agnostic Multilingual Streaming On-Device ASR Systemcited by 9
16 more
  1. 2021Multi-View Self-Attention Based Transformer for Speaker Recognitioncited by 8
  2. 2024An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglementcited by 8
  3. 2022Scaling Up Deliberation for Multilingual ASRcited by 6
  4. 2025Efficient Multilingual ASR Finetuning via LoRA Language Expertscited by 5
  5. 2021Multilingual Speech Recognition using Knowledge Transfer across Learning Processescited by 4
  6. 2023Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Trainingcited by 4
  7. 2023UML: A Universal Monolingual Output Layer for Multilingual ASRcited by 3
  8. 2023Confidence-based Ensembles of End-to-End Speech Recognition Modelscited by 3
  9. 2024Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Promptingcited by 3
  10. 2024Reverb: Open-Source ASR and Diarization from Revcited by 3
  11. 2023Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognitioncited by 2
  12. 2024Improving Zero-Shot Chinese-English Code-Switching ASR with kNN-CTC and Gated Monolingual Datastorescited by 2
  13. 2023Simple yet Effective Code-Switching Language Identification with Multitask Pre-Training and Transfer Learningcited by 1
  14. 2025Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTCcited by 1
  15. 2024Configurable Multilingual ASR with Speech Summary Representationscited by 0
  16. 2026Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusioncited by 0