# Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding

arXiv: https://arxiv.org/abs/2412.16507 · HTML: https://mlsanity.com/paper/2412.16507/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 21 Dec 2024.

## Builds on (most cited first, 30 of 35)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization](https://mlsanity.com/paper/2210.14567/) (2022)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Enhancing Code-switching Speech Recognition with Interactive Language Biases](https://mlsanity.com/paper/2309.16953/) (2023)
- [Language-specific Characteristic Assistance for Code-switching Speech Recognition](https://mlsanity.com/paper/2206.14580/) (2022)
- [Aligning Speech to Languages to Enhance Code-switching Speech Recognition](https://mlsanity.com/paper/2403.05887/) (2024)
- [Fine-grained Disentangled Representation Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2312.13567/) (2023)
- [Rethinking Mamba in Speech Processing by Self-Supervised Models](https://mlsanity.com/paper/2409.07273/) (2024)
- [Attention-Guided Adaptation for Code-Switching Speech Recognition](https://mlsanity.com/paper/2312.08856/) (2023)
- [Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models](https://mlsanity.com/paper/2110.03504/) (2021)
- [MERLIon CCS Challenge: A English-Mandarin code-switching child-directed speech corpus for language identification and diarization](https://mlsanity.com/paper/2305.18881/) (2023)
- [Adapting OpenAI's Whisper for Speech Recognition on Code-Switch Mandarin-English SEAME and ASRU2019 Datasets](https://mlsanity.com/paper/2311.17382/) (2023)
- [An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement](https://mlsanity.com/paper/2402.17189/) (2024)
- [Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition](https://mlsanity.com/paper/2409.00815/) (2024)
- [LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR](https://mlsanity.com/paper/2309.16178/) (2023)
- [Adapting the adapters for code-switching in multilingual ASR](https://mlsanity.com/paper/2310.07423/) (2023)
- [Internal Language Model Estimation based Language Model Fusion for Cross-Domain Code-Switching Speech Recognition](https://mlsanity.com/paper/2207.04176/) (2022)
- [An Adapter based Multi-label Pre-training for Speech Separation and Enhancement](https://mlsanity.com/paper/2211.06041/) (2022)
- [A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition](https://mlsanity.com/paper/2311.07093/) (2023)
- [Exploration of Adapter for Noise Robust Automatic Speech Recognition](https://mlsanity.com/paper/2402.18275/) (2024)

## Built on by (most cited first, 7 of 7)

- [Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognition](https://mlsanity.com/paper/2509.04488/) (2025)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark](https://mlsanity.com/paper/2602.12911/) (2026)
- [Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs](https://mlsanity.com/paper/2605.23975/) (2026)
- [AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition](https://mlsanity.com/paper/2608.26434/) (2026)
- [Automatic Speech Recognition for Multilingual Oral History Research](https://mlsanity.com/paper/2609.04232/) (2026)
- [Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition](https://mlsanity.com/paper/2609.27289/) (2026)

## Abstract (arXiv)

Code-switching (CS) automatic speech recognition (ASR) faces challenges due to the language confusion resulting from accents, auditory similarity, and seamless language switches. Adaptation on the pre-trained multi-lingual model has shown promising performance for CS-ASR. In this paper, we adapt Whisper, which is a large-scale multilingual pre-trained speech recognition model, to CS from both encoder and decoder parts. First, we propose an encoder refiner to enhance the encoder's capacity of intra-sentence swithching. Second, we propose using two sets of language-aware adapters with different language prompt embeddings to achieve language-specific decoding information in each decoder layer. Then, a fusion module is added to fuse the language-aware decoding. The experimental results using the SEAME dataset show that, compared with the baseline model, the proposed approach achieves a relative MER reduction of 4.1% and 7.2% on the dev_man and dev_sge test sets, respectively, surpassing state-of-the-art methods. Through experiments, we found that the proposed method significantly improves the performance on non-native language in CS speech, indicating that our approach enables Whisper to better distinguish between the two languages.