# Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models

arXiv: https://arxiv.org/abs/2110.03504 · HTML: https://mlsanity.com/paper/2110.03504/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Vector-Quantized Autoregressive Predictive Coding](https://mlsanity.com/paper/2005.08392/) (2020)
- [Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies](https://mlsanity.com/paper/2011.00406/) (2020)
- [On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/1811.00241/) (2018)
- [Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition](https://mlsanity.com/paper/2006.10414/) (2020)
- [Transformer-Transducers for Code-Switched Speech Recognition](https://mlsanity.com/paper/2011.15023/) (2020)
- [Code-switching Sentence Generation by Generative Adversarial Networks and its Application to Data Augmentation](https://mlsanity.com/paper/1811.02356/) (2018)
- [Data Augmentation for End-to-end Code-switching Speech Recognition](https://mlsanity.com/paper/2011.02160/) (2020)
- [Improving Low Resource Code-switched ASR using Augmented Code-switched TTS](https://mlsanity.com/paper/2010.05549/) (2020)
- [Rnn-transducer with language bias for end-to-end Mandarin-English code-switching speech recognition](https://mlsanity.com/paper/2002.08126/) (2020)
- [Non-autoregressive Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/2104.02258/) (2021)
- [Utterance-level end-to-end language identification using attention-based CNN-BLSTM](https://mlsanity.com/paper/1902.07374/) (2019)

## Built on by (most cited first, 9 of 9)

- [Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering](https://mlsanity.com/paper/2305.11072/) (2023)
- [Code-Switching Text Generation and Injection in Mandarin-English ASR](https://mlsanity.com/paper/2303.10949/) (2023)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [Adapting the adapters for code-switching in multilingual ASR](https://mlsanity.com/paper/2310.07423/) (2023)
- [Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data](https://mlsanity.com/paper/2407.10603/) (2024)
- [ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models](https://mlsanity.com/paper/2604.10065/) (2026)
- [Monolingual Recognizers Fusion for Code-switching Speech Recognition](https://mlsanity.com/paper/2211.01046/) (2022)
- [Adapting Language Balance in Code-Switching Speech](https://mlsanity.com/paper/2510.18724/) (2025)
- [The Exact Worst-Case Tail Probability under Bounded Kurtosis](https://mlsanity.com/paper/2607.05226/) (2026)

## Abstract (arXiv)

Code-switching (CS) is common in daily conversations where more than one language is used within a sentence. The difficulties of CS speech recognition lie in alternating languages and the lack of transcribed data. Therefore, this paper uses the recently successful self-supervised learning (SSL) methods to leverage many unlabeled speech data without CS. We show that hidden representations of SSL models offer frame-level language identity even if the models are trained with English speech only. Jointly training CTC and language identification modules with self-supervised speech representations improves CS speech recognition performance. Furthermore, using multilingual speech data for pre-training obtains the best CS speech recognition.