# Efficient Training of Neural Transducer for Speech Recognition

arXiv: https://arxiv.org/abs/2204.10586 · HTML: https://mlsanity.com/paper/2204.10586/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 28 of 28)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [SpecAugment on Large Scale Datasets](https://mlsanity.com/paper/1912.05533/) (2019)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [A New Training Pipeline for an Improved Neural Transducer](https://mlsanity.com/paper/2005.09319/) (2020)
- [Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognition](https://mlsanity.com/paper/2007.13802/) (2020)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging](https://mlsanity.com/paper/2012.06749/) (2020)
- [Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models](https://mlsanity.com/paper/2104.05544/) (2021)
- [Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition](https://mlsanity.com/paper/1911.12487/) (2019)
- [Phoneme Based Neural Transducer for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/2010.16368/) (2020)
- [The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment](https://mlsanity.com/paper/2004.00960/) (2020)
- [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Acoustic Data-Driven Subword Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/2104.09106/) (2021)

## Built on by (most cited first, 14 of 14)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition](https://mlsanity.com/paper/2407.18930/) (2024)
- [Enhancing and Adversarial: Improve ASR with Speaker Labels](https://mlsanity.com/paper/2211.06369/) (2022)
- [Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems](https://mlsanity.com/paper/2306.13307/) (2023)
- [Comparative Analysis of the wav2vec 2.0 Feature Extractor](https://mlsanity.com/paper/2308.04286/) (2023)
- [Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers](https://mlsanity.com/paper/2212.04325/) (2022)
- [RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognition](https://mlsanity.com/paper/2305.17782/) (2023)
- [Regularizing Learnable Feature Extraction for Automatic Speech Recognition](https://mlsanity.com/paper/2506.09804/) (2025)
- [On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducers](https://mlsanity.com/paper/2309.14130/) (2023)
- [End-to-End Training of a Neural HMM with Label and Transition Probabilities](https://mlsanity.com/paper/2310.02724/) (2023)
- [Dynamic Acoustic Model Architecture Optimization in Training for ASR](https://mlsanity.com/paper/2506.13180/) (2025)
- [Analyzing And Improving Neural Speaker Embeddings for ASR](https://mlsanity.com/paper/2301.04571/) (2023)
- [Investigating the Effect of Language Models in Sequence Discriminative Training for Neural Transducers](https://mlsanity.com/paper/2310.07345/) (2023)
- [Unified Learnable 2D Convolutional Feature Extraction for ASR](https://mlsanity.com/paper/2509.10031/) (2025)

## Abstract (arXiv)

As one of the most popular sequence-to-sequence modeling approaches for speech recognition, the RNN-Transducer has achieved evolving performance with more and more sophisticated neural network models of growing size and increasing training epochs. While strong computation resources seem to be the prerequisite of training superior models, we try to overcome it by carefully designing a more efficient training pipeline. In this work, we propose an efficient 3-stage progressive training pipeline to build highly-performing neural transducer models from scratch with very limited computation resources in a reasonable short time period. The effectiveness of each stage is experimentally verified on both Librispeech and Switchboard corpora. The proposed pipeline is able to train transducer models approaching state-of-the-art performance with a single GPU in just 2-3 weeks. Our best conformer transducer achieves 4.1% WER on Librispeech test-other with only 35 epochs of training.