# Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization

arXiv: https://arxiv.org/abs/2210.14567 · HTML: https://mlsanity.com/paper/2210.14567/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2022.

## Built on by (most cited first, 29 of 29)

- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition](https://mlsanity.com/paper/2402.05457/) (2024)
- [Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision](https://mlsanity.com/paper/2401.00273/) (2023)
- [Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages](https://mlsanity.com/paper/2310.03018/) (2023)
- [An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement](https://mlsanity.com/paper/2402.17189/) (2024)
- [Generative error correction for code-switching speech recognition using large language models](https://mlsanity.com/paper/2310.13013/) (2023)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR](https://mlsanity.com/paper/2309.16178/) (2023)
- [The Second DISPLACE Challenge : DIarization of SPeaker and LAnguage in Conversational Environments](https://mlsanity.com/paper/2406.09494/) (2024)
- [SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR](https://mlsanity.com/paper/2406.18021/) (2024)
- [Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages](https://mlsanity.com/paper/2506.14177/) (2025)
- [Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning](https://mlsanity.com/paper/2506.21576/) (2025)
- [Summary of the DISPLACE Challenge 2023 - DIarization of SPeaker and LAnguage in Conversational Environments](https://mlsanity.com/paper/2311.12564/) (2023)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition](https://mlsanity.com/paper/2412.12760/) (2024)
- [Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing](https://mlsanity.com/paper/2407.18581/) (2024)
- [AdaCS: Adaptive Normalization for Enhanced Code-Switching ASR](https://mlsanity.com/paper/2501.07102/) (2025)
- [Improving Code-Switching Speech Recognition with TTS Data Augmentation](https://mlsanity.com/paper/2601.00935/) (2026)
- [Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs](https://mlsanity.com/paper/2605.23975/) (2026)
- [Romanization Encoding For Multilingual ASR](https://mlsanity.com/paper/2407.04368/) (2024)
- [Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech](https://mlsanity.com/paper/2606.19381/) (2026)
- [Leveraging Language ID to Calculate Intermediate CTC Loss for Enhanced Code-Switching Speech Recognition](https://mlsanity.com/paper/2312.09583/) (2023)
- [Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection](https://mlsanity.com/paper/2412.08651/) (2024)
- [SAGE-LD: Towards Scalable and Generalizable End-to-End Language Diarization via Simulated Data Augmentation](https://mlsanity.com/paper/2510.00582/) (2025)
- [Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration](https://mlsanity.com/paper/2601.02906/) (2026)
- [Reinforcement Learning for Data-Efficient Code-Switched ASR](https://mlsanity.com/paper/2607.02757/) (2026)
- [The Exact Worst-Case Tail Probability under Bounded Kurtosis](https://mlsanity.com/paper/2607.05226/) (2026)
- [Code-Switching Spoken Language Identification as Multi-Label Set Prediction](https://mlsanity.com/paper/2610.01450/) (2026)
- [LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR](https://mlsanity.com/paper/2610.16579/) (2026)

## Abstract (arXiv)

Code-switching (CS) refers to the phenomenon that languages switch within a speech signal and leads to language confusion for automatic speech recognition (ASR). This paper aims to address language confusion for improving CS-ASR from two perspectives: incorporating and disentangling language information. We incorporate language information in the CS-ASR model by dynamically biasing the model with token-level language posteriors which are outputs of a sequence-to-sequence auxiliary language diarization module. In contrast, the disentangling process reduces the difference between languages via adversarial training so as to normalize two languages. We conduct the experiments on the SEAME dataset. Compared to the baseline model, both the joint optimization with LD and the language posterior bias achieve performance improvement. The comparison of the proposed methods indicates that incorporating language information is more effective than disentangling for reducing language confusion in CS speech.