# Language-specific Characteristic Assistance for Code-switching Speech Recognition

arXiv: https://arxiv.org/abs/2206.14580 · HTML: https://mlsanity.com/paper/2206.14580/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 29 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [A Survey of Code-switched Speech and Language Processing](https://mlsanity.com/paper/1904.00784/) (2019)
- [Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2012.05481/) (2020)
- [On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/1811.00241/) (2018)
- [Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition](https://mlsanity.com/paper/2006.10414/) (2020)
- [Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization](https://mlsanity.com/paper/2111.15016/) (2021)
- [Learning to Recognize Code-switched Speech Without Forgetting Monolingual Speech Recognition](https://mlsanity.com/paper/2006.00782/) (2020)
- [Constrained Output Embeddings for End-to-End Code-Switching Speech Recognition with Only Monolingual Data](https://mlsanity.com/paper/1904.03802/) (2019)
- [Decoupling Pronunciation and Language for End-to-end Code-switching Automatic Speech Recognition](https://mlsanity.com/paper/2010.14798/) (2020)

## Built on by (most cited first, 20 of 20)

- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition](https://mlsanity.com/paper/2307.05956/) (2023)
- [Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning](https://mlsanity.com/paper/2212.05301/) (2022)
- [Towards Zero-Shot Code-Switched Speech Recognition](https://mlsanity.com/paper/2211.01458/) (2022)
- [BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition](https://mlsanity.com/paper/2310.02629/) (2023)
- [An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement](https://mlsanity.com/paper/2402.17189/) (2024)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR](https://mlsanity.com/paper/2309.16178/) (2023)
- [SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR](https://mlsanity.com/paper/2406.18021/) (2024)
- [Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation](https://mlsanity.com/paper/2210.12214/) (2022)
- [Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages](https://mlsanity.com/paper/2506.14177/) (2025)
- [Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognition](https://mlsanity.com/paper/2509.04488/) (2025)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition](https://mlsanity.com/paper/2412.12760/) (2024)
- [Monolingual Recognizers Fusion for Code-switching Speech Recognition](https://mlsanity.com/paper/2211.01046/) (2022)
- [Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing](https://mlsanity.com/paper/2407.18581/) (2024)
- [Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model](https://mlsanity.com/paper/2409.02050/) (2024)
- [Developing vocal system impaired patient-aimed voice quality assessment approach using ASR representation-included multiple features](https://mlsanity.com/paper/2408.12279/) (2024)
- [Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection](https://mlsanity.com/paper/2412.08651/) (2024)
- [The Exact Worst-Case Tail Probability under Bounded Kurtosis](https://mlsanity.com/paper/2607.05226/) (2026)

## Abstract (arXiv)

Dual-encoder structure successfully utilizes two language-specific encoders (LSEs) for code-switching speech recognition. Because LSEs are initialized by two pre-trained language-specific models (LSMs), the dual-encoder structure can exploit sufficient monolingual data and capture the individual language attributes. However, most existing methods have no language constraints on LSEs and underutilize language-specific knowledge of LSMs. In this paper, we propose a language-specific characteristic assistance (LSCA) method to mitigate the above problems. Specifically, during training, we introduce two language-specific losses as language constraints and generate corresponding language-specific targets for them. During decoding, we take the decoding abilities of LSMs into account by combining the output probabilities of two LSMs and the mixture model to obtain the final predictions. Experiments show that either the training or decoding method of LSCA can improve the model's performance. Furthermore, the best result can obtain up to 15.4% relative error reduction on the code-switching test set by combining the training and decoding methods of LSCA. Moreover, the system can process code-switching speech recognition tasks well without extra shared parameters or even retraining based on two pre-trained LSMs by using our method.