mlsanityData to 7 Oct 2026

SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts

arXiv 2105.03036 ↗First posted 7 May 2021cs.SD cs.CL eess.AScited by 48

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Recently, Mixture of Experts (MoE) based Transformer has shown promising results in many domains. This is largely due to the following advantages of this architecture: firstly, MoE based Transformer can increase model capacity without computational cost increasing both at training and inference time. Besides, MoE based Transformer is a dynamic network which can adapt to the varying complexity of input instances in realworld applications. In this work, we explore the MoE based model for speech recognition, named SpeechMoE. To further control the sparsity of router activation and improve the diversity of gate values, we propose a sparsity L1 loss and a mean importance loss respectively. In addition, a new router architecture is used in SpeechMoE which can simultaneously utilize the information from a shared embedding network and the hierarchical representation of different MoE layers. Experimental results show that SpeechMoE can achieve lower character error rate (CER) with comparable computation cost than traditional static networks, providing 7.0%-23.0% relative CER improvements on four evaluation datasets.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 14 of its 25 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

14 of 25 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  8. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  9. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  10. 2018Light Gated Recurrent Units for Speech Recognitioncited by 48
  11. 2018Deep-FSMN for Large Vocabulary Continuous Speech Recognitioncited by 28
  12. 2018Recent Progresses in Deep Learning based Acoustic Models (Updated)cited by 8
2 more
  1. 2019DFSMN-SAN with Persistent Memory Model for Automatic Speech Recognitioncited by 4
  2. 2016Very Deep Convolutional Neural Networks for Robust Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 13; 2024: 8; 2025: 13; 2026: 4.

48 papers.

Sort:

By area:

  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  2. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  3. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  4. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  5. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  6. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
  7. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  8. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  9. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognitioncited by 26
  10. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  11. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  12. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
36 more
  1. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  2. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  3. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  4. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16
  5. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  6. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  7. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Expertscited by 14
  8. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  9. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  10. 2023Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inferencecited by 9
  11. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settingscited by 7
  12. 2023LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASRcited by 6
  13. 20223M: Multi-loss, Multi-path and Multi-level Neural Networks for speech recognitioncited by 5
  14. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTFcited by 5
  15. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  16. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  17. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  18. 2022Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognitioncited by 3
  19. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
  20. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  21. 2024TouchASP: Elastic Automatic Speech Perception that Everyone Can Touchcited by 2
  22. 2025Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challengescited by 2
  23. 2025Mixture of Low-Rank Adapter Experts in Generalizable Audio Deepfake Detectioncited by 2
  24. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  25. 2022Summary on the ISCSLP 2022 Chinese-English Code-Switching ASR Challengecited by 1
  26. 2023ASR Bundestag: A Large-Scale political debate dataset in Germancited by 1
  27. 2024UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognitioncited by 1
  28. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  29. 2023One-stop Training of Multiple Capacity Modelscited by 0
  30. 2023Direct Neural Machine Translation with Task-level Mixture of Experts modelscited by 0
  31. 2025On-the-fly Routing for Zero-shot MoE Speaker Adaptation of Speech Foundation Models for Dysarthric Speech Recognitioncited by 0
  32. 2025Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognitioncited by 0
  33. 2026Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognitioncited by 0
  34. 2026Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASRcited by 0
  35. 2026MME: Mixture of Mesh Experts with Random Walk Transformer Gatingcited by 0
  36. 2026DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models
  2. 2022StableMoE: Stable Routing Strategy for Mixture of Experts
  3. 2024Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing
  4. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
  5. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  6. 2026TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation
  7. 2024Layerwise Recurrent Router for Mixture-of-Experts
  8. 2025Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts