# A Token-Wise Beam Search Algorithm for RNN-T

arXiv: https://arxiv.org/abs/2302.14357 · HTML: https://mlsanity.com/paper/2302.14357/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [A comparison of end-to-end models for long-form speech recognition](https://mlsanity.com/paper/1911.02242/) (2019)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation](https://mlsanity.com/paper/2110.05249/) (2021)
- [Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging](https://mlsanity.com/paper/2012.06749/) (2020)
- [Scaling ASR Improves Zero and Few Shot Learning](https://mlsanity.com/paper/2111.05948/) (2021)
- [A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies](https://mlsanity.com/paper/2201.05420/) (2022)
- [Improving Spoken Language Understanding By Exploiting ASR N-best Hypotheses](https://mlsanity.com/paper/2001.05284/) (2020)
- [Factorized Blank Thresholding for Improved Runtime Efficiency of Neural Transducers](https://mlsanity.com/paper/2211.00896/) (2022)
- [An Investigation of Monotonic Transducers for Large-Scale Automatic Speech Recognition](https://mlsanity.com/paper/2204.08858/) (2022)

## Built on by (most cited first, 3 of 3)

- [Pushing the Limits of Beam Search Decoding for Transducer-based ASR models](https://mlsanity.com/paper/2506.00185/) (2025)
- [Token-Weighted RNN-T for Learning from Flawed Data](https://mlsanity.com/paper/2406.18108/) (2024)
- [WIND: Accelerated RNN-T Decoding with Windowed Inference for Non-blank Detection](https://mlsanity.com/paper/2505.13765/) (2025)

## Abstract (arXiv)

Standard Recurrent Neural Network Transducers (RNN-T) decoding algorithms for speech recognition are iterating over the time axis, such that one time step is decoded before moving on to the next time step. Those algorithms result in a large number of calls to the joint network, which were shown in previous work to be an important factor that reduces decoding speed. We present a decoding beam search algorithm that batches the joint network calls across a segment of time steps, which results in 20%-96% decoding speedups consistently across all models and settings experimented with. In addition, aggregating emission probabilities over a segment may be seen as a better approximation to finding the most likely model output, causing our algorithm to improve oracle word error rate by up to 11% relative as the segment size increases, and to slightly improve general word error rate.