mlsanityData to 7 Oct 2026

Mixture-of-Expert Conformer for Streaming Multilingual ASR

arXiv 2305.15663 ↗First posted 25 May 2023cs.CL cs.SD eess.AScited by 17

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

End-to-end models with large capacity have significantly improved multilingual automatic speech recognition, but their computation cost poses challenges for on-device applications. We propose a streaming truly multilingual Conformer incorporating mixture-of-expert (MoE) layers that learn to only activate a subset of parameters in training and inference. The MoE layer consists of a softmax gate which chooses the best two experts among many in forward propagation. The proposed MoE layer offers efficient inference by activating a fixed number of parameters as the number of experts increases. We evaluate the proposed model on a set of 12 languages, and achieve an average 11.9% relative improvement in WER over the baseline. Compared to an adapter model using ground truth information, our MoE model achieves similar WER and activates similar number of parameters but without any language information. We further show around 3% relative WER improvement by multilingual shallow fusion.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 24 of its 34 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

24 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routing23 Nov
  2. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May

Other references · 22

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  10. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  11. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  12. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
10 more
  1. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 51
  2. 2021A Configurable Multilingual Model is All You Need to Recognize All Languagescited by 28
  3. 2018Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languagescited by 26
  4. 2022Streaming End-to-End Multilingual Speech Recognition with Joint Language Identificationcited by 24
  5. 2017Towards Language-Universal End-to-End Speech Recognitioncited by 14
  6. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  7. 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
  8. 2023Massively Multilingual Shallow Fusion with Large Language Modelscited by 10
  9. 2022A Language Agnostic Multilingual Streaming On-Device ASR Systemcited by 9
  10. 2022Scaling Up Deliberation for Multilingual ASRcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 4; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASRcited by 6
  2. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTFcited by 5
  3. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  4. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  5. 2024Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASRcited by 3
  6. 2023Big model only for hard audios: Sample dependent Whisper model selection for efficient inferencescited by 2
  7. 2024Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routingcited by 2
  8. 2024Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Modelcited by 2
  9. 2024TouchASP: Elastic Automatic Speech Perception that Everyone Can Touchcited by 2
  10. 2024UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognitioncited by 1
  11. 2025SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challengecited by 1
  12. 2026A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASRcited by 1
5 more
  1. 2024Configurable Multilingual ASR with Speech Summary Representationscited by 0
  2. 2025Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognitioncited by 0
  3. 2026MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgettingcited by 0
  4. 2026Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognitioncited by 0
  5. 2026Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgettingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition
  2. 2026Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR
  3. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
  4. 2026Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition
  5. 2022Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition
  6. 2024LUPET: Incorporating Hierarchical Information Path into Multilingual ASR
  7. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition
  8. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling