# Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languages

arXiv: https://arxiv.org/abs/1806.05059 · HTML: https://mlsanity.com/paper/1806.05059/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2018.

## Built on by (most cited first, 26 of 26)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability](https://mlsanity.com/paper/2308.03266/) (2023)
- [Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition](https://mlsanity.com/paper/2101.06699/) (2021)
- [Mixture-of-Expert Conformer for Streaming Multilingual ASR](https://mlsanity.com/paper/2305.15663/) (2023)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)
- [A Language Agnostic Multilingual Streaming On-Device ASR System](https://mlsanity.com/paper/2208.13916/) (2022)
- [Improved Meta Learning for Low Resource Speech Recognition](https://mlsanity.com/paper/2205.06182/) (2022)
- [A Novel Self-training Approach for Low-resource Speech Recognition](https://mlsanity.com/paper/2308.05269/) (2023)
- [Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning](https://mlsanity.com/paper/2208.03067/) (2022)
- [Scaling Up Deliberation for Multilingual ASR](https://mlsanity.com/paper/2210.05785/) (2022)
- [OLR 2021 Challenge: Datasets, Rules and Baselines](https://mlsanity.com/paper/2107.11113/) (2021)
- [Huqariq: A Multilingual Speech Corpus of Native Languages of Peru for Speech Recognition](https://mlsanity.com/paper/2207.05498/) (2022)
- [A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learning](https://mlsanity.com/paper/2305.13331/) (2023)
- [Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios](https://mlsanity.com/paper/2104.02232/) (2021)
- [Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting](https://mlsanity.com/paper/2406.12611/) (2024)
- [Language-universal phonetic encoder for low-resource speech recognition](https://mlsanity.com/paper/2305.11576/) (2023)
- [A Dual-Decoder Conformer for Multilingual Speech Recognition](https://mlsanity.com/paper/2109.03277/) (2021)
- [Transfer Learning for Multi-lingual Tasks -- a Survey](https://mlsanity.com/paper/2110.02052/) (2021)
- [Integrating Categorical Features in End-to-End ASR](https://mlsanity.com/paper/2110.03047/) (2021)
- [Towards Building an End-to-End Multilingual Automatic Lyrics Transcription Model](https://mlsanity.com/paper/2406.17618/) (2024)
- [Multilingual Speech Recognition for Low-Resource Indian Languages using Multi-Task conformer](https://mlsanity.com/paper/2109.03969/) (2021)
- [Objective Soups: Multilingual Multi-Task Modeling for Speech Processing](https://mlsanity.com/paper/2508.09228/) (2025)
- [Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languages](https://mlsanity.com/paper/2606.09535/) (2026)

## Abstract (arXiv)

Sequence-to-sequence attention-based models integrate an acoustic, pronunciation and language model into a single neural network, which make them very suitable for multilingual automatic speech recognition (ASR). In this paper, we are concerned with multilingual speech recognition on low-resource languages by a single Transformer, one of sequence-to-sequence attention-based models. Sub-words are employed as the multilingual modeling unit without using any pronunciation lexicon. First, we show that a single multilingual ASR Transformer performs well on low-resource languages despite of some language confusion. We then look at incorporating language information into the model by inserting the language symbol at the beginning or at the end of the original sub-words sequence under the condition of language information being known during training. Experiments on CALLHOME datasets demonstrate that the multilingual ASR Transformer with the language symbol at the end performs better and can obtain relatively 10.5% average word error rate (WER) reduction compared to SHL-MLSTM with residual learning. We go on to show that, assuming the language information being known during training and testing, about relatively 12.4% average WER reduction can be observed compared to SHL-MLSTM with residual learning through giving the language symbol as the sentence start token.