# Multilingual Speech Recognition With A Single End-To-End Model

arXiv: https://arxiv.org/abs/1711.01694 · HTML: https://mlsanity.com/paper/1711.01694/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2017.

## Built on by (most cited first, 30 of 87)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR](https://mlsanity.com/paper/2105.14779/) (2021)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement](https://mlsanity.com/paper/2406.11546/) (2024)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [Improving Continuous Sign Language Recognition with Cross-Lingual Signs](https://mlsanity.com/paper/2308.10809/) (2023)
- [Pseudo-Labeling for Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2111.00161/) (2021)
- [Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification](https://mlsanity.com/paper/2209.06058/) (2022)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)
- [Arabic Code-Switching Speech Recognition using Monolingual Data](https://mlsanity.com/paper/2107.01573/) (2021)
- [Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2109.09161/) (2021)
- [Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition](https://mlsanity.com/paper/2112.05820/) (2021)
- [Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech](https://mlsanity.com/paper/2108.12881/) (2021)
- [ArTST: Arabic Text and Speech Transformer](https://mlsanity.com/paper/2310.16621/) (2023)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability](https://mlsanity.com/paper/2211.02499/) (2022)
- [Multilingual and crosslingual speech recognition using phonological-vector based phone embeddings](https://mlsanity.com/paper/2107.05038/) (2021)
- [Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation](https://mlsanity.com/paper/2204.02470/) (2022)
- [Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning](https://mlsanity.com/paper/2306.15686/) (2023)
- [Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity](https://mlsanity.com/paper/2111.01326/) (2021)
- [WER We Stand: Benchmarking Urdu ASR Models](https://mlsanity.com/paper/2409.11252/) (2024)
- [Distilling a Pretrained Language Model to a Multilingual ASR Model](https://mlsanity.com/paper/2206.12638/) (2022)
- [ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems](https://mlsanity.com/paper/2102.08535/) (2021)
- [Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning](https://mlsanity.com/paper/2208.03067/) (2022)
- [SpeakerStew: Scaling to Many Languages with a Triaged Multilingual Text-Dependent and Text-Independent Speaker Verification System](https://mlsanity.com/paper/2104.02125/) (2021)

## Abstract (arXiv)

Training a conventional automatic speech recognition (ASR) system to support multiple languages is challenging because the sub-word unit, lexicon and word inventories are typically language specific. In contrast, sequence-to-sequence models are well suited for multilingual ASR because they encapsulate an acoustic, pronunciation and language model jointly in a single network. In this work we present a single sequence-to-sequence ASR model trained on 9 different Indian languages, which have very little overlap in their scripts. Specifically, we take a union of language-specific grapheme sets and train a grapheme-based sequence-to-sequence model jointly on data from all languages. We find that this model, which is not explicitly given any information about language identity, improves recognition performance by 21% relative compared to analogous sequence-to-sequence models trained on each language individually. By modifying the model to accept a language identifier as an additional input feature, we further improve performance by an additional 7% relative and eliminate confusion between different languages.