# Adapting the adapters for code-switching in multilingual ASR

arXiv: https://arxiv.org/abs/2310.07423 · HTML: https://mlsanity.com/paper/2310.07423/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation](https://mlsanity.com/paper/2112.06223/) (2021)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [QASR: QCRI Aljazeera Speech Resource -- A Large Scale Annotated Arabic Speech Corpus](https://mlsanity.com/paper/2106.13000/) (2021)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Arabic Code-Switching Speech Recognition using Monolingual Data](https://mlsanity.com/paper/2107.01573/) (2021)
- [End-to-End Code-Switching ASR for Low-Resourced Language Pairs](https://mlsanity.com/paper/1909.12681/) (2019)
- [Building a Unified Code-Switching ASR System for South African Languages](https://mlsanity.com/paper/1807.10949/) (2018)
- [Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models](https://mlsanity.com/paper/2110.03504/) (2021)
- [Code Switched and Code Mixed Speech Recognition for Indic languages](https://mlsanity.com/paper/2203.16578/) (2022)

## Built on by (most cited first, 6 of 6)

- [Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision](https://mlsanity.com/paper/2401.00273/) (2023)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [The Balancing Act: Unmasking and Alleviating ASR Biases in Portuguese](https://mlsanity.com/paper/2402.07513/) (2024)
- [SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR](https://mlsanity.com/paper/2406.18021/) (2024)
- [A Survey of Code-switched Arabic NLP: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2501.13419/) (2025)
- [Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs](https://mlsanity.com/paper/2606.05846/) (2026)

## Abstract (arXiv)

Recently, large pre-trained multilingual speech models have shown potential in scaling Automatic Speech Recognition (ASR) to many low-resource languages. Some of these models employ language adapters in their formulation, which helps to improve monolingual performance and avoids some of the drawbacks of multi-lingual modeling on resource-rich languages. However, this formulation restricts the usability of these models on code-switched speech, where two languages are mixed together in the same utterance. In this work, we propose ways to effectively fine-tune such models on code-switched speech, by assimilating information from both language adapters at each language adaptation point in the network. We also model code-switching as a sequence of latent binary sequences that can be used to guide the flow of information from each language adapter at the frame level. The proposed approaches are evaluated on three code-switched datasets encompassing Arabic, Mandarin, and Hindi languages paired with English, showing consistent improvements in code-switching performance with at least 10% absolute reduction in CER across all test sets.