# Streaming Bilingual End-to-End ASR model using Attention over Multiple Softmax

arXiv: https://arxiv.org/abs/2401.11645 · HTML: https://mlsanity.com/paper/2401.11645/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition](https://mlsanity.com/paper/2005.14327/) (2020)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [Transfer Learning for Speech Recognition on a Budget](https://mlsanity.com/paper/1706.00290/) (2017)
- [Streaming End-to-End Bilingual ASR Systems with Joint Language Identification](https://mlsanity.com/paper/2007.03900/) (2020)
- [Highway Long Short-Term Memory RNNs for Distant Speech Recognition](https://mlsanity.com/paper/1510.08983/) (2015)
- [Transfer Learning Approaches for Streaming End-to-End Speech Recognition System](https://mlsanity.com/paper/2008.05086/) (2020)
- [Bootstrap an end-to-end ASR system by multilingual training, transfer learning, text-to-text mapping and synthetic audio](https://mlsanity.com/paper/2011.12696/) (2020)

## Built on by (most cited first, 1 of 1)

- [Building English ASR model with regional language support](https://mlsanity.com/paper/2503.07522/) (2025)

## Abstract (arXiv)

Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availability of the input language. In this work, we propose a novel bilingual end-to-end (E2E) modeling approach, where a single neural model can recognize both languages and also support switching between the languages, without any language input from the user. The proposed model has shared encoder and prediction networks, with language-specific joint networks that are combined via a self-attention mechanism. As the language-specific posteriors are combined, it produces a single posterior probability over all the output symbols, enabling a single beam search decoding and also allowing dynamic switching between the languages. The proposed approach outperforms the conventional bilingual baseline with 13.3%, 8.23% and 1.3% word error rate relative reduction on Hindi, English and code-mixed test sets, respectively.