# SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR

arXiv: https://arxiv.org/abs/2406.18021 · HTML: https://mlsanity.com/paper/2406.18021/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 20 of 20)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results](https://mlsanity.com/paper/2007.05916/) (2020)
- [MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognition](https://mlsanity.com/paper/2302.13750/) (2023)
- [U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2106.05642/) (2021)
- [From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text](https://mlsanity.com/paper/2107.06483/) (2021)
- [Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization](https://mlsanity.com/paper/2210.14567/) (2022)
- [Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition](https://mlsanity.com/paper/2006.10414/) (2020)
- [Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition](https://mlsanity.com/paper/2307.05956/) (2023)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Data Augmentation for End-to-end Code-switching Speech Recognition](https://mlsanity.com/paper/2011.02160/) (2020)
- [Language-specific Characteristic Assistance for Code-switching Speech Recognition](https://mlsanity.com/paper/2206.14580/) (2022)
- [Mixture-of-Expert Conformer for Streaming Multilingual ASR](https://mlsanity.com/paper/2305.15663/) (2023)
- [Code-Switching Text Generation and Injection in Mandarin-English ASR](https://mlsanity.com/paper/2303.10949/) (2023)
- [Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching](https://mlsanity.com/paper/2311.15077/) (2023)
- [BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition](https://mlsanity.com/paper/2310.02629/) (2023)
- [LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR](https://mlsanity.com/paper/2309.16178/) (2023)
- [Adapting the adapters for code-switching in multilingual ASR](https://mlsanity.com/paper/2310.07423/) (2023)

## Built on by (most cited first, 6 of 6)

- [PIER: A Novel Metric for Evaluating What Matters in Code-Switching](https://mlsanity.com/paper/2501.09512/) (2025)
- [Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM](https://mlsanity.com/paper/2409.15905/) (2024)
- [CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition](https://mlsanity.com/paper/2412.12760/) (2024)
- [Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs](https://mlsanity.com/paper/2605.23975/) (2026)
- [Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling](https://mlsanity.com/paper/2606.10439/) (2026)
- [SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge](https://mlsanity.com/paper/2507.03343/) (2025)

## Abstract (arXiv)

In this work, we propose a Switch-Conformer-based MoE system named SC-MoE for unified streaming and non-streaming code-switching (CS) automatic speech recognition (ASR), where we design a streaming MoE layer consisting of three language experts, which correspond to Mandarin, English, and blank, respectively, and equipped with a language identification (LID) network with a Connectionist Temporal Classification (CTC) loss as a router in the encoder of SC-MoE to achieve a real-time streaming CS ASR system. To further utilize the language information embedded in text, we also incorporate MoE layers into the decoder of SC-MoE. In addition, we introduce routers into every MoE layer of the encoder and the decoder and achieve better recognition performance. Experimental results show that the SC-MoE significantly improves CS ASR performances over baseline with comparable computational efficiency.