# Towards Zero-Shot Code-Switched Speech Recognition

arXiv: https://arxiv.org/abs/2211.01458 · HTML: https://mlsanity.com/paper/2211.01458/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 30 of 32)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results](https://mlsanity.com/paper/2007.05916/) (2020)
- [Joint Unsupervised and Supervised Training for Multilingual ASR](https://mlsanity.com/paper/2111.08137/) (2021)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/1811.00241/) (2018)
- [First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs](https://mlsanity.com/paper/1408.2873/) (2014)
- [Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition](https://mlsanity.com/paper/2006.10414/) (2020)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [Transformer-Transducers for Code-Switched Speech Recognition](https://mlsanity.com/paper/2011.15023/) (2020)
- [ASR2K: Speech Recognition for Around 2000 Languages without Audio](https://mlsanity.com/paper/2209.02842/) (2022)
- [Pseudo-Labeling for Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2111.00161/) (2021)
- [Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification](https://mlsanity.com/paper/2209.06058/) (2022)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Language-specific Characteristic Assistance for Code-switching Speech Recognition](https://mlsanity.com/paper/2206.14580/) (2022)
- [Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization](https://mlsanity.com/paper/2111.15016/) (2021)
- [Arabic Code-Switching Speech Recognition using Monolingual Data](https://mlsanity.com/paper/2107.01573/) (2021)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training](https://mlsanity.com/paper/1810.11895/) (2018)
- [Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networks](https://mlsanity.com/paper/2203.04583/) (2022)
- [Learning to Recognize Code-switched Speech Without Forgetting Monolingual Speech Recognition](https://mlsanity.com/paper/2006.00782/) (2020)
- [Differentiable Allophone Graphs for Language-Universal Speech Recognition](https://mlsanity.com/paper/2107.11628/) (2021)

## Built on by (most cited first, 15 of 15)

- [PIER: A Novel Metric for Evaluating What Matters in Code-Switching](https://mlsanity.com/paper/2501.09512/) (2025)
- [Speech collage: code-switched audio generation by collaging monolingual corpora](https://mlsanity.com/paper/2309.15674/) (2023)
- [BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition](https://mlsanity.com/paper/2310.02629/) (2023)
- [An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement](https://mlsanity.com/paper/2402.17189/) (2024)
- [Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition](https://mlsanity.com/paper/2309.11327/) (2023)
- [Generative error correction for code-switching speech recognition using large language models](https://mlsanity.com/paper/2310.13013/) (2023)
- [LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR](https://mlsanity.com/paper/2309.16178/) (2023)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [Improving Zero-Shot Chinese-English Code-Switching ASR with kNN-CTC and Gated Monolingual Datastores](https://mlsanity.com/paper/2406.03814/) (2024)
- [Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing](https://mlsanity.com/paper/2407.18581/) (2024)
- [Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model](https://mlsanity.com/paper/2409.02050/) (2024)
- [Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking](https://mlsanity.com/paper/2409.18428/) (2024)
- [Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection](https://mlsanity.com/paper/2412.08651/) (2024)
- [Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration](https://mlsanity.com/paper/2601.02906/) (2026)
- [The Exact Worst-Case Tail Probability under Bounded Kurtosis](https://mlsanity.com/paper/2607.05226/) (2026)

## Abstract (arXiv)

In this work, we seek to build effective code-switched (CS) automatic speech recognition systems (ASR) under the zero-shot setting where no transcribed CS speech data is available for training. Previously proposed frameworks which conditionally factorize the bilingual task into its constituent monolingual parts are a promising starting point for leveraging monolingual data efficiently. However, these methods require the monolingual modules to perform language segmentation. That is, each monolingual module has to simultaneously detect CS points and transcribe speech segments of one language while ignoring those of other languages -- not a trivial task. We propose to simplify each monolingual module by allowing them to transcribe all speech segments indiscriminately with a monolingual script (i.e. transliteration). This simple modification passes the responsibility of CS point detection to subsequent bilingual modules which determine the final output by considering multiple monolingual transliterations along with external language model information. We apply this transliteration-based approach in an end-to-end differentiable neural network and demonstrate its efficacy for zero-shot CS ASR on Mandarin-English SEAME test sets.