# Advancing RNN Transducer Technology for Speech Recognition

arXiv: https://arxiv.org/abs/2103.09935 · HTML: https://mlsanity.com/paper/2103.09935/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 21 of 21)

- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://mlsanity.com/paper/1808.07512/) (2018)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Exploring Neural Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1707.07413/) (2017)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [A New Training Pipeline for an Improved Neural Transducer](https://mlsanity.com/paper/2005.09319/) (2020)
- [RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions](https://mlsanity.com/paper/2005.03271/) (2020)
- [Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition](https://mlsanity.com/paper/1911.12487/) (2019)
- [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019)
- [On Multiplicative Integration with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06630/) (2016)
- [Exploring RNN-Transducer for Chinese Speech Recognition](https://mlsanity.com/paper/1811.05097/) (2018)

## Built on by (most cited first, 30 of 44)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [Large-Scale Streaming End-to-End Speech Translation with Neural Transducers](https://mlsanity.com/paper/2204.05352/) (2022)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems](https://mlsanity.com/paper/2204.00212/) (2022)
- [Integrating Text Inputs For Training and Adapting RNN Transducer ASR Models](https://mlsanity.com/paper/2202.13155/) (2022)
- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [Accelerating Transducers through Adjacent Token Merging](https://mlsanity.com/paper/2306.16009/) (2023)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability](https://mlsanity.com/paper/2211.02499/) (2022)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognition](https://mlsanity.com/paper/2110.03841/) (2021)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [Improving the Training Recipe for a Robust Conformer-based Hybrid Model](https://mlsanity.com/paper/2206.12955/) (2022)
- [Recurrent Aligned Network for Generalized Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2403.05810/) (2024)
- [Brain-inspired Artificial Intelligence: A Comprehensive Review](https://mlsanity.com/paper/2408.14811/) (2024)
- [4-bit Quantization of LSTM-based Speech Recognition Models](https://mlsanity.com/paper/2108.12074/) (2021)
- [Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding](https://mlsanity.com/paper/2204.05169/) (2022)
- [RNN Transducer Models For Spoken Language Understanding](https://mlsanity.com/paper/2104.03842/) (2021)
- [CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition](https://mlsanity.com/paper/2307.14132/) (2023)
- [Semi-Autoregressive Streaming ASR With Label Context](https://mlsanity.com/paper/2309.10926/) (2023)
- [Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers](https://mlsanity.com/paper/2107.03007/) (2021)
- [A Deep Reinforcement Learning Blind AI in DareFightingICE](https://mlsanity.com/paper/2205.07444/) (2022)
- [Improving RNN-Transducers with Acoustic LookAhead](https://mlsanity.com/paper/2307.05006/) (2023)
- [Boosting keyword spotting through on-device learnable user speech characteristics](https://mlsanity.com/paper/2403.07802/) (2024)
- [HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2506.17587/) (2025)

## Abstract (arXiv)

We investigate a set of techniques for RNN Transducers (RNN-Ts) that were instrumental in lowering the word error rate on three different tasks (Switchboard 300 hours, conversational Spanish 780 hours and conversational Italian 900 hours). The techniques pertain to architectural changes, speaker adaptation, language model fusion, model combination and general training recipe. First, we introduce a novel multiplicative integration of the encoder and prediction network vectors in the joint network (as opposed to additive). Second, we discuss the applicability of i-vector speaker adaptation to RNN-Ts in conjunction with data perturbation. Third, we explore the effectiveness of the recently proposed density ratio language model fusion for these tasks. Last but not least, we describe the other components of our training recipe and their effect on recognition performance. We report a 5.9% and 12.5% word error rate on the Switchboard and CallHome test sets of the NIST Hub5 2000 evaluation and a 12.7% WER on the Mozilla CommonVoice Italian test set.