mlsanityData to 14 Oct 2026

SpeechMoE2: Mixture-of-Experts Model with Improved Routing

arXiv 2111.11831 ↗First posted 23 Nov 2021eess.AS cs.CL cs.SDcited by 26

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Mixture-of-experts based acoustic models with dynamic routing mechanisms have proved promising results for speech recognition. The design principle of router architecture is important for the large model capacity and high computational efficiency. Our previous work SpeechMoE only uses local grapheme embedding to help routers to make route decisions. To further improve speech recognition performance against varying domains and accents, we propose a new router architecture which integrates additional global domain and accent embedding into router input to promote adaptability. Experimental results show that the proposed SpeechMoE2 can achieve lower character error rate (CER) with comparable parameters than SpeechMoE on both multi-domain and multi-accent task. Primarily, the proposed method provides up to 1.6% - 4.8% relative CER improvement for the multidomain task and 1.9% - 17.7% relative CER improvement for the multi-accent task respectively. Besides, increasing the number of experts also achieves consistent performance improvement and keeps the computational cost constant.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 9 of its 15 references that are in the map. First posted on arXiv 23 Nov 2021.

Builds on

9 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May

Other references · 8

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  5. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 100
  6. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  7. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  8. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 10; 2024: 4; 2025: 5; 2026: 2.

26 papers.

Sort:

By area:

  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  2. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  3. 2023Modular Deep Learningcited by 83
  4. 2024Fast On-device LLM Inference with NPUscited by 62
  5. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  6. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
  7. 2022Sparsity-Constrained Optimal Transportcited by 32
  8. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 29
  9. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  10. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  11. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  12. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
14 more
  1. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16
  2. 20223M: Multi-loss, Multi-path and Multi-level Neural Networks for speech recognitioncited by 5
  3. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  4. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  5. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  6. 2024Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routingcited by 2
  7. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  8. 2023ASR Bundestag: A Large-Scale political debate dataset in Germancited by 1
  9. 2024UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognitioncited by 1
  10. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  11. 2023Direct Neural Machine Translation with Task-level Mixture of Experts modelscited by 0
  12. 2025Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognitioncited by 0
  13. 2026Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertscited by 0
  14. 2026Towards Generalization-Oriented Models for Vehicle Routing Problems with Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models
  2. 2022StableMoE: Stable Routing Strategy for Mixture of Experts
  3. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition
  4. 2022Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition
  5. 2024Routers in Vision Mixture of Experts: An Empirical Study
  6. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
  7. 2024HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models
  8. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models