# Dual Script E2E framework for Multilingual and Code-Switching ASR

arXiv: https://arxiv.org/abs/2106.01400 · HTML: https://mlsanity.com/paper/2106.01400/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 18 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [Language-agnostic Multilingual Modeling](https://mlsanity.com/paper/2004.09571/) (2020)
- [Generic Indic Text-to-speech Synthesisers with Rapid Adaptation in an End-to-end Framework](https://mlsanity.com/paper/2006.06971/) (2020)

## Built on by (most cited first, 6 of 6)

- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [An Overview of Indian Spoken Language Recognition from Machine Learning Perspective](https://mlsanity.com/paper/2212.03812/) (2022)
- [UniCoM: A Universal Code-Switching Speech Generator](https://mlsanity.com/paper/2508.15244/) (2025)
- [DuDe: Dual-Decoder Multilingual ASR for Indian Languages using Common Label Set](https://mlsanity.com/paper/2210.16739/) (2022)
- [When Multiple Scripts Matter: Evaluating ASR in Clinical Settings](https://mlsanity.com/paper/2606.17826/) (2026)

## Abstract (arXiv)

India is home to multiple languages, and training automatic speech recognition (ASR) systems for languages is challenging. Over time, each language has adopted words from other languages, such as English, leading to code-mixing. Most Indian languages also have their own unique scripts, which poses a major limitation in training multilingual and code-switching ASR systems. Inspired by results in text-to-speech synthesis, in this work, we use an in-house rule-based phoneme-level common label set (CLS) representation to train multilingual and code-switching ASR for Indian languages. We propose two end-to-end (E2E) ASR systems. In the first system, the E2E model is trained on the CLS representation, and we use a novel data-driven back-end to recover the native language script. In the second system, we propose a modification to the E2E model, wherein the CLS representation and the native language characters are used simultaneously for training. We show our results on the multilingual and code-switching tasks of the Indic ASR Challenge 2021. Our best results achieve 6% and 5% improvement (approx) in word error rate over the baseline system for the multilingual and code-switching tasks, respectively, on the challenge development data.