mlsanityData to 7 Oct 2026

Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition

arXiv 2112.05820 ↗First posted 10 Dec 2021cs.CL cs.AI cs.LGcited by 16

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity with a little computational complexity. In this work, we investigate how multi-lingual Automatic Speech Recognition (ASR) networks can be scaled up with a simple routing algorithm in order to achieve better accuracy. More specifically, we apply the sparsely-gated MoE technique to two types of networks: Sequence-to-Sequence Transformer (S2S-T) and Transformer Transducer (T-T). We demonstrate through a set of ASR experiments on multiple language data that the MoE networks can reduce the relative word error rates by 16.3% and 4.6% with the S2S-T and T-T, respectively. Moreover, we thoroughly investigate the effect of the MoE on the T-T architecture in various conditions: streaming mode, non-streaming mode, the use of language ID and the label decoder with the MoE.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 19 of its 34 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

19 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routing23 Nov
  2. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May

Other references · 17

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  5. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  6. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  7. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  8. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  9. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  10. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  11. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  12. 2021A Configurable Multilingual Model is All You Need to Recognize All Languagescited by 28
5 more
  1. 2020Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASRcited by 19
  2. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 15
  3. 2017Towards Language-Universal End-to-End Speech Recognitioncited by 14
  4. 2021UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Datasetcited by 7
  5. 2021Multilingual Speech Recognition using Knowledge Transfer across Learning Processescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 3; 2025: 4; 2026: 1.

16 papers.

Sort:

By area:

  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  2. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  3. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  4. 2023Modular Deep Learningcited by 82
  5. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  6. 2024Multi-Head Mixture-of-Expertscited by 30
  7. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognitioncited by 26
  8. 2022MoEC: Mixture of Expert Clusterscited by 14
  9. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  10. 2025Tight Clusters Make Specialized Expertscited by 8
  11. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  12. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
4 more
  1. 2024MH-MoE: Multi-Head Mixture-of-Expertscited by 4
  2. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structurescited by 3
  3. 2022Inference and Denoise: Causal Inference-based Neural Speech Enhancementcited by 0
  4. 2026TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASR
  2. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
  3. 2022Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition
  4. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF
  5. 2026Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition
  6. 2024Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing
  7. 2024Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model
  8. 2025Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts