# An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement

arXiv: https://arxiv.org/abs/2402.17189 · HTML: https://mlsanity.com/paper/2402.17189/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 19 of 19)

- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Self-training and Pre-training are Complementary for Speech Recognition](https://mlsanity.com/paper/2010.11430/) (2020)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results](https://mlsanity.com/paper/2007.05916/) (2020)
- [Improving Massively Multilingual ASR With Auxiliary CTC Objectives](https://mlsanity.com/paper/2302.12829/) (2023)
- [On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/1811.00241/) (2018)
- [Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization](https://mlsanity.com/paper/2210.14567/) (2022)
- [Massively Multilingual Adversarial Speech Recognition](https://mlsanity.com/paper/1904.02210/) (2019)
- [Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition](https://mlsanity.com/paper/2006.10414/) (2020)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [Transformer-Transducers for Code-Switched Speech Recognition](https://mlsanity.com/paper/2011.15023/) (2020)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Language-specific Characteristic Assistance for Code-switching Speech Recognition](https://mlsanity.com/paper/2206.14580/) (2022)
- [Towards End-to-End Code-Switching Speech Recognition](https://mlsanity.com/paper/1810.13091/) (2018)
- [Towards Zero-Shot Code-Switched Speech Recognition](https://mlsanity.com/paper/2211.01458/) (2022)
- [Minimum word error training for non-autoregressive Transformer-based code-switching ASR](https://mlsanity.com/paper/2110.03573/) (2021)
- [MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition](https://mlsanity.com/paper/2212.00500/) (2022)
- [Integrating Knowledge in End-to-End Automatic Speech Recognition for Mandarin-English Code-Switching](https://mlsanity.com/paper/2112.10202/) (2021)

## Built on by (most cited first, 8 of 8)

- [PIER: A Novel Metric for Evaluating What Matters in Code-Switching](https://mlsanity.com/paper/2501.09512/) (2025)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages](https://mlsanity.com/paper/2506.14177/) (2025)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition](https://mlsanity.com/paper/2412.12760/) (2024)
- [Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing](https://mlsanity.com/paper/2407.18581/) (2024)
- [Adapting Language Balance in Code-Switching Speech](https://mlsanity.com/paper/2510.18724/) (2025)
- [Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection](https://mlsanity.com/paper/2412.08651/) (2024)

## Abstract (arXiv)

With the massive developments of end-to-end (E2E) neural networks, recent years have witnessed unprecedented breakthroughs in automatic speech recognition (ASR). However, the codeswitching phenomenon remains a major obstacle that hinders ASR from perfection, as the lack of labeled data and the variations between languages often lead to degradation of ASR performance. In this paper, we focus exclusively on improving the acoustic encoder of E2E ASR to tackle the challenge caused by the codeswitching phenomenon. Our main contributions are threefold: First, we introduce a novel disentanglement loss to enable the lower-layer of the encoder to capture inter-lingual acoustic information while mitigating linguistic confusion at the higher-layer of the encoder. Second, through comprehensive experiments, we verify that our proposed method outperforms the prior-art methods using pretrained dual-encoders, meanwhile having access only to the codeswitching corpus and consuming half of the parameterization. Third, the apparent differentiation of the encoders' output features also corroborates the complementarity between the disentanglement loss and the mixture-of-experts (MoE) architecture.