mlsanityData to 14 Oct 2026

Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition

arXiv 2507.05724 ↗First posted 8 Jul 2025cs.CL cs.AI cs.LGcited by 4

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Mixture-of-experts (MoE) architectures have expanded from language modeling to automatic speech recognition (ASR). Traditional MoE methods, such as the Switch Transformer, route experts independently within each layer. Our analysis reveals that routers in most layers make expert choices that are not strongly correlated with the choices of the routers in other layers. To increase the cooperation between experts in different layers and encourage greater specialization, we use a shared router across different MoE layers. We call this model Omni-router Transformer. Extensive experiments on a large-scale pseudo-labeled dataset and evaluations across 10 diverse, out-of-domain ASR benchmarks demonstrate that the Omni-router Transformer is able to achieve lower training loss and consistently outperform dense and Switch Transformer models, reducing average word error rates by 11.2% and 8.2%, respectively, while providing structured expert usage and improved robustness to diverse data.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 42 of its 54 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

42 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASR25 May
  2. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routing23 Nov
  3. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
27 more
  1. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  4. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  5. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  7. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  9. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  10. 2021Hash Layers For Large Sparse Modelscited by 237
  11. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  12. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  13. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  14. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  15. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  16. 2019A spelling correction model for end-to-end speech recognitioncited by 63
  17. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  18. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  19. 2024Transformer Layers as Painterscited by 54
  20. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  21. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  22. 2018Improved training of end-to-end attention models for speech recognitioncited by 36
  23. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  24. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 26
  25. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 21
  26. 2018Fully Convolutional Speech Recognitioncited by 19
  27. 2025OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Modelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025ChipChat: Low-Latency Cascaded Conversational Agent in MLXcited by 4
  2. 2026Path-Constrained Mixture-of-Expertscited by 4
  3. 2025Segmental Attention Decoding With Long Form Acoustic Encodingscited by 0
  4. 2026Hierarchical Prompt-Domain Control and Learning for Resource-Constrained Agentic Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Routers in Vision Mixture of Experts: An Empirical Study
  2. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition
  3. 2024Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models
  4. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
  5. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  6. 2026MoRE: Mixture of Reused Experts
  7. 2026Mixture of Experts for Low-Resource LLMs
  8. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF