# Exploring Transformers for Large-Scale Speech Recognition

arXiv: https://arxiv.org/abs/2005.09684 · HTML: https://mlsanity.com/paper/2005.09684/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 24 of 24)

- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [End-to-End Multi-Channel Transformer for Speech Recognition](https://mlsanity.com/paper/2102.03951/) (2021)
- [Wake Word Detection with Streaming Transformers](https://mlsanity.com/paper/2102.04488/) (2021)
- [Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR](https://mlsanity.com/paper/2304.04974/) (2023)
- [Multi-Channel Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2108.12953/) (2021)
- [Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2309.15796/) (2023)
- [Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR](https://mlsanity.com/paper/2102.06816/) (2021)
- [Exploring the Performance and Efficiency of Transformer Models for NLP on Mobile Devices](https://mlsanity.com/paper/2306.11426/) (2023)
- [Munsit at NADI 2025 Shared Task 2: Pushing the Boundaries of Multidialectal Arabic ASR with Weakly Supervised Pretraining and Continual Supervised Fine-tuning](https://mlsanity.com/paper/2508.08912/) (2025)
- [Bridging the Granularity Gap for Acoustic Modeling](https://mlsanity.com/paper/2305.17356/) (2023)
- [Advancing Arabic Speech Recognition Through Large-Scale Weakly Supervised Learning](https://mlsanity.com/paper/2504.12254/) (2025)
- [Beyond Isolated Utterances: Conversational Emotion Recognition](https://mlsanity.com/paper/2109.06112/) (2021)
- [WST: Weakly Supervised Transducer for Automatic Speech Recognition](https://mlsanity.com/paper/2511.04035/) (2025)
- [DEFORMER: Coupling Deformed Localized Patterns with Global Context for Robust End-to-end Speech Recognition](https://mlsanity.com/paper/2207.01732/) (2022)
- [Deep Sparse Conformer for Speech Recognition](https://mlsanity.com/paper/2209.00260/) (2022)
- [Towards Personalization of CTC Speech Recognition Models with Contextual Adapters and Adaptive Boosting](https://mlsanity.com/paper/2210.09510/) (2022)
- [Accelerator-Aware Training for Transducer-Based Speech Recognition](https://mlsanity.com/paper/2305.07778/) (2023)
- [Bilingual Streaming ASR with Grapheme units and Auxiliary Monolingual Loss](https://mlsanity.com/paper/2308.06327/) (2023)
- [Revisiting Interpolation Augmentation for Speech-to-Text Generation](https://mlsanity.com/paper/2406.15846/) (2024)
- [A multilevel approach to accelerate the training of Transformers](https://mlsanity.com/paper/2504.18590/) (2025)
- [Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)](https://mlsanity.com/paper/2508.09957/) (2025)

## Abstract (arXiv)

While recurrent neural networks still largely define state-of-the-art speech recognition systems, the Transformer network has been proven to be a competitive alternative, especially in the offline condition. Most studies with Transformers have been constrained in a relatively small scale setting, and some forms of data argumentation approaches are usually applied to combat the data sparsity issue. In this paper, we aim at understanding the behaviors of Transformers in the large-scale speech recognition setting, where we have used around 65,000 hours of training data. We investigated various aspects on scaling up Transformers, including model initialization, warmup training as well as different Layer Normalization strategies. In the streaming condition, we compared the widely used attention mask based future context lookahead approach to the Transformer-XL network. From our experiments, we show that Transformers can achieve around 6% relative word error rate (WER) reduction compared to the BLSTM baseline in the offline fashion, while in the streaming fashion, Transformer-XL is comparable to LC-BLSTM with 800 millisecond latency constraint.