# Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition

arXiv: https://arxiv.org/abs/2006.10414 · HTML: https://mlsanity.com/paper/2006.10414/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 28 of 28)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR](https://mlsanity.com/paper/2105.14779/) (2021)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Language-specific Characteristic Assistance for Code-switching Speech Recognition](https://mlsanity.com/paper/2206.14580/) (2022)
- [Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization](https://mlsanity.com/paper/2111.15016/) (2021)
- [Towards Zero-Shot Code-Switched Speech Recognition](https://mlsanity.com/paper/2211.01458/) (2022)
- [Code-Switching Text Generation and Injection in Mandarin-English ASR](https://mlsanity.com/paper/2303.10949/) (2023)
- [Speech collage: code-switched audio generation by collaging monolingual corpora](https://mlsanity.com/paper/2309.15674/) (2023)
- [Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models](https://mlsanity.com/paper/2110.03504/) (2021)
- [BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition](https://mlsanity.com/paper/2310.02629/) (2023)
- [Non-autoregressive Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/2104.02258/) (2021)
- [An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement](https://mlsanity.com/paper/2402.17189/) (2024)
- [SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR](https://mlsanity.com/paper/2406.18021/) (2024)
- [Internal Language Model Estimation based Language Model Fusion for Cross-Domain Code-Switching Speech Recognition](https://mlsanity.com/paper/2207.04176/) (2022)
- [Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data](https://mlsanity.com/paper/2407.10603/) (2024)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [Monolingual Recognizers Fusion for Code-switching Speech Recognition](https://mlsanity.com/paper/2211.01046/) (2022)
- [Balanced End-to-End Monolingual pre-training for Low-Resourced Indic Languages Code-Switching Speech Recognition](https://mlsanity.com/paper/2106.05885/) (2021)
- [Revisiting Context Choices for Context-aware Machine Translation](https://mlsanity.com/paper/2109.02995/) (2021)
- [Romanization Encoding For Multilingual ASR](https://mlsanity.com/paper/2407.04368/) (2024)
- [XCB: an effective contextual biasing approach to bias cross-lingual phrases in speech recognition](https://mlsanity.com/paper/2408.10524/) (2024)
- [Computational Approaches to Arabic-English Code-Switching](https://mlsanity.com/paper/2410.13318/) (2024)
- [Building Korean Sign Language Augmentation (KoSLA) Corpus with Data Augmentation Technique](https://mlsanity.com/paper/2207.05261/) (2022)
- [Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection](https://mlsanity.com/paper/2412.08651/) (2024)
- [The Exact Worst-Case Tail Probability under Bounded Kurtosis](https://mlsanity.com/paper/2607.05226/) (2026)

## Abstract (arXiv)

Code-switching (CS) occurs when a speaker alternates words of two or more languages within a single sentence or across sentences. Automatic speech recognition (ASR) of CS speech has to deal with two or more languages at the same time. In this study, we propose a Transformer-based architecture with two symmetric language-specific encoders to capture the individual language attributes, that improve the acoustic representation of each language. These representations are combined using a language-specific multi-head attention mechanism in the decoder module. Each encoder and its corresponding attention module in the decoder are pre-trained using a large monolingual corpus aiming to alleviate the impact of limited CS training data. We call such a network a multi-encoder-decoder (MED) architecture. Experiments on the SEAME corpus show that the proposed MED architecture achieves 10.2% and 10.8% relative error rate reduction on the CS evaluation sets with Mandarin and English as the matrix language respectively.