# Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2005.00572 · HTML: https://mlsanity.com/paper/2005.00572/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 9 of 9)

- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Multi-blank Transducers for Speech Recognition](https://mlsanity.com/paper/2211.03541/) (2022)
- [SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory](https://mlsanity.com/paper/2108.13630/) (2021)
- [Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Mask](https://mlsanity.com/paper/2110.04056/) (2021)
- [Memory-Efficient Training of RNN-Transducer with Sampled Softmax](https://mlsanity.com/paper/2203.16868/) (2022)
- [HMM-Free Encoder Pre-Training for Streaming RNN Transducer](https://mlsanity.com/paper/2104.10764/) (2021)
- [StableEmit: Selection Probability Discount for Reducing Emission Latency of Streaming Monotonic Attention ASR](https://mlsanity.com/paper/2107.00635/) (2021)
- [Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals](https://mlsanity.com/paper/2308.08125/) (2023)
- [Variational Connectionist Temporal Classification for Order-Preserving Sequence Modeling](https://mlsanity.com/paper/2309.11983/) (2023)

## Abstract (arXiv)

Recently, the recurrent neural network transducer (RNN-T) architecture has become an emerging trend in end-to-end automatic speech recognition research due to its advantages of being capable for online streaming speech recognition. However, RNN-T training is made difficult by the huge memory requirements, and complicated neural structure. A common solution to ease the RNN-T training is to employ connectionist temporal classification (CTC) model along with RNN language model (RNNLM) to initialize the RNN-T parameters. In this work, we conversely leverage external alignments to seed the RNN-T model. Two different pre-training solutions are explored, referred to as encoder pre-training, and whole-network pre-training respectively. Evaluated on Microsoft 65,000 hours anonymized production data with personally identifiable information removed, our proposed methods can obtain significant improvement. In particular, the encoder pre-training solution achieved a 10% and a 8% relative word error rate reduction when compared with random initialization and the widely used CTC+RNNLM initialization strategy, respectively. Our solutions also significantly reduce the RNN-T model latency from the baseline.