# Towards Language-Universal End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/1711.02207 · HTML: https://mlsanity.com/paper/1711.02207/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2017.

## Built on by (most cited first, 14 of 14)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification](https://mlsanity.com/paper/2209.06058/) (2022)
- [Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization](https://mlsanity.com/paper/2111.15016/) (2021)
- [Arabic Code-Switching Speech Recognition using Monolingual Data](https://mlsanity.com/paper/2107.01573/) (2021)
- [Mixture-of-Expert Conformer for Streaming Multilingual ASR](https://mlsanity.com/paper/2305.15663/) (2023)
- [Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition](https://mlsanity.com/paper/2112.05820/) (2021)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)
- [Reducing language context confusion for end-to-end code-switching automatic speech recognition](https://mlsanity.com/paper/2201.12155/) (2022)
- [Scaling Up Deliberation for Multilingual ASR](https://mlsanity.com/paper/2210.05785/) (2022)
- [UML: A Universal Monolingual Output Layer for Multilingual ASR](https://mlsanity.com/paper/2302.11186/) (2023)
- [Semi-supervised transfer learning for language expansion of end-to-end speech recognition models to low-resource languages](https://mlsanity.com/paper/2111.10047/) (2021)
- [Language-universal phonetic encoder for low-resource speech recognition](https://mlsanity.com/paper/2305.11576/) (2023)
- [Coarse-To-Fine And Cross-Lingual ASR Transfer](https://mlsanity.com/paper/2109.00916/) (2021)

## Abstract (arXiv)

Building speech recognizers in multiple languages typically involves replicating a monolingual training recipe for each language, or utilizing a multi-task learning approach where models for different languages have separate output labels but share some internal parameters. In this work, we exploit recent progress in end-to-end speech recognition to create a single multilingual speech recognition system capable of recognizing any of the languages seen in training. To do so, we propose the use of a universal character set that is shared among all languages. We also create a language-specific gating mechanism within the network that can modulate the network's internal representations in a language-specific way. We evaluate our proposed approach on the Microsoft Cortana task across three languages and show that our system outperforms both the individual monolingual systems and systems built with a multi-task learning approach. We also show that this model can be used to initialize a monolingual speech recognizer, and can be used to create a bilingual model for use in code-switching scenarios.