# Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition

arXiv: https://arxiv.org/abs/2307.05956 · HTML: https://mlsanity.com/paper/2307.05956/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results](https://mlsanity.com/paper/2007.05916/) (2020)
- [Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM](https://mlsanity.com/paper/1706.02737/) (2017)
- [MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognition](https://mlsanity.com/paper/2302.13750/) (2023)
- [Transformer-Transducers for Code-Switched Speech Recognition](https://mlsanity.com/paper/2011.15023/) (2020)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Language-specific Characteristic Assistance for Code-switching Speech Recognition](https://mlsanity.com/paper/2206.14580/) (2022)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization](https://mlsanity.com/paper/2111.15016/) (2021)
- [Towards End-to-End Code-Switching Speech Recognition](https://mlsanity.com/paper/1810.13091/) (2018)
- [Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition](https://mlsanity.com/paper/2112.05820/) (2021)
- [Leveraging Phone Mask Training for Phonetic-Reduction-Robust E2E Uyghur Speech Recognition](https://mlsanity.com/paper/2204.00819/) (2022)
- [Multilingual Speech Recognition using Knowledge Transfer across Learning Processes](https://mlsanity.com/paper/2110.07909/) (2021)
- [PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition](https://mlsanity.com/paper/2112.06721/) (2021)

## Built on by (most cited first, 26 of 26)

- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR](https://mlsanity.com/paper/2309.16178/) (2023)
- [The Second DISPLACE Challenge : DIarization of SPeaker and LAnguage in Conversational Environments](https://mlsanity.com/paper/2406.09494/) (2024)
- [SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR](https://mlsanity.com/paper/2406.18021/) (2024)
- [Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation](https://mlsanity.com/paper/2502.17380/) (2025)
- [Efficient Multilingual ASR Finetuning via LoRA Language Experts](https://mlsanity.com/paper/2506.21555/) (2025)
- [U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF](https://mlsanity.com/paper/2404.16407/) (2024)
- [Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages](https://mlsanity.com/paper/2506.14177/) (2025)
- [On the Role of Discrete Representation in Sparse Mixture of Experts](https://mlsanity.com/paper/2411.19402/) (2024)
- [MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition](https://mlsanity.com/paper/2502.10447/) (2025)
- [SimSMoE: Solving Representational Collapse via Similarity Measure](https://mlsanity.com/paper/2406.15883/) (2024)
- [TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch](https://mlsanity.com/paper/2412.15622/) (2024)
- [Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing](https://mlsanity.com/paper/2407.18581/) (2024)
- [Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model](https://mlsanity.com/paper/2409.02050/) (2024)
- [BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR](https://mlsanity.com/paper/2501.12602/) (2025)
- [Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR](https://mlsanity.com/paper/2506.21577/) (2025)
- [Adapting Language Balance in Code-Switching Speech](https://mlsanity.com/paper/2510.18724/) (2025)
- [UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition](https://mlsanity.com/paper/2412.17507/) (2024)
- [SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge](https://mlsanity.com/paper/2507.03343/) (2025)
- [A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASR](https://mlsanity.com/paper/2601.00557/) (2026)
- [Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision](https://mlsanity.com/paper/2603.07025/) (2026)
- [Configurable Multilingual ASR with Speech Summary Representations](https://mlsanity.com/paper/2410.04478/) (2024)
- [Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition](https://mlsanity.com/paper/2509.08470/) (2025)
- [MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2607.24030/) (2026)
- [RouteRec: Behavior-Guided Sparse Routing for Sequential Recommendation](https://mlsanity.com/paper/2609.39007/) (2026)

## Abstract (arXiv)

Multilingual speech recognition for both monolingual and code-switching speech is a challenging task. Recently, based on the Mixture of Experts (MoE), many works have made good progress in multilingual and code-switching ASR, but present huge computational complexity with the increase of supported languages. In this work, we propose a computation-efficient network named Language-Routing Mixture of Experts (LR-MoE) for multilingual and code-switching ASR. LR-MoE extracts language-specific representations through the Mixture of Language Experts (MLE), which is guided to learn by a frame-wise language routing mechanism. The weight-shared frame-level language identification (LID) network is jointly trained as the shared pre-router of each MoE layer. Experiments show that the proposed method significantly improves multilingual and code-switching speech recognition performances over baseline with comparable computational efficiency.