# Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer

arXiv: https://arxiv.org/abs/1801.00841 · HTML: https://mlsanity.com/paper/1801.00841/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2018.

## Built on by (most cited first, 30 of 96)

- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition](https://mlsanity.com/paper/2109.01163/) (2021)
- [Factorized Neural Transducer for Efficient Language Model Adaptation](https://mlsanity.com/paper/2110.01500/) (2021)
- [Black-box Adversarial Attacks on Commercial Speech Platforms with Minimal Information](https://mlsanity.com/paper/2110.09714/) (2021)
- [Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks](https://mlsanity.com/paper/2109.14320/) (2021)
- [Integrating Text Inputs For Training and Adapting RNN Transducer ASR Models](https://mlsanity.com/paper/2202.13155/) (2022)
- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network](https://mlsanity.com/paper/2104.11127/) (2021)
- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability](https://mlsanity.com/paper/2308.03266/) (2023)
- [Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities](https://mlsanity.com/paper/2211.05756/) (2022)
- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025)
- [Multi-Head State Space Model for Speech Recognition](https://mlsanity.com/paper/2305.12498/) (2023)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [Decoder-only Architecture for Streaming End-to-end Speech Recognition](https://mlsanity.com/paper/2406.16107/) (2024)
- [Echo State Speech Recognition](https://mlsanity.com/paper/2102.09114/) (2021)
- [FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization](https://mlsanity.com/paper/2104.02882/) (2021)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Code-Switching Text Generation and Injection in Mandarin-English ASR](https://mlsanity.com/paper/2303.10949/) (2023)
- [Understanding Medical Conversations: Rich Transcription, Confidence Scores & Information Extraction](https://mlsanity.com/paper/2104.02219/) (2021)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency](https://mlsanity.com/paper/2104.02176/) (2021)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [Internal Language Model Estimation Through Explicit Context Vector Learning for Attention-based Encoder-decoder ASR](https://mlsanity.com/paper/2201.11627/) (2022)
- [Cross-Speaker Encoding Network for Multi-Talker Speech Recognition](https://mlsanity.com/paper/2401.04152/) (2024)

## Abstract (arXiv)

We investigate training end-to-end speech recognition models with the recurrent neural network transducer (RNN-T): a streaming, all-neural, sequence-to-sequence architecture which jointly learns acoustic and language model components from transcribed acoustic data. We explore various model architectures and demonstrate how the model can be improved further if additional text or pronunciation data are available. The model consists of an `encoder', which is initialized from a connectionist temporal classification-based (CTC) acoustic model, and a `decoder' which is partially initialized from a recurrent neural network language model trained on text data alone. The entire neural network is trained with the RNN-T loss and directly outputs the recognized transcript as a sequence of graphemes, thus performing end-to-end speech recognition. We find that performance can be improved further through the use of sub-word units (`wordpieces') which capture longer context and significantly reduce substitution errors. The best RNN-T system, a twelve-layer LSTM encoder with a two-layer LSTM decoder trained with 30,000 wordpieces as output targets achieves a word error rate of 8.5% on voice-search and 5.2% on voice-dictation tasks and is comparable to a state-of-the-art baseline at 8.3% on voice-search and 5.4% voice-dictation.