# Pushing the Limits of Beam Search Decoding for Transducer-based ASR models

arXiv: https://arxiv.org/abs/2506.00185 · HTML: https://mlsanity.com/paper/2506.00185/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 18 of 18)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition](https://mlsanity.com/paper/2104.02014/) (2021)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019)
- [Accelerating RNN-T Training and Inference Using CTC guidance](https://mlsanity.com/paper/2210.16481/) (2022)
- [Modular Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2210.17049/) (2022)
- [Accelerating RNN Transducer Inference via One-Step Constrained Beam Search](https://mlsanity.com/paper/2002.03577/) (2020)
- [Fast and parallel decoding for transducer](https://mlsanity.com/paper/2211.00484/) (2022)
- [Speed of Light Exact Greedy Decoding for RNN-T Speech Recognition Models on GPU](https://mlsanity.com/paper/2406.03791/) (2024)
- [Label-Looping: Highly Efficient Decoding for Transducers](https://mlsanity.com/paper/2406.06220/) (2024)
- [Streaming parallel transducer beam search with fast-slow cascaded encoders](https://mlsanity.com/paper/2203.15773/) (2022)
- [Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spotter](https://mlsanity.com/paper/2406.07096/) (2024)
- [NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding](https://mlsanity.com/paper/2505.22857/) (2025)
- [A Token-Wise Beam Search Algorithm for RNN-T](https://mlsanity.com/paper/2302.14357/) (2023)

## Built on by (most cited first, 4 of 4)

- [TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Tree](https://mlsanity.com/paper/2508.07014/) (2025)
- [FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilities](https://mlsanity.com/paper/2508.07315/) (2025)
- [MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation task](https://mlsanity.com/paper/2606.17255/) (2026)
- [TurboBias 2.0: Streaming Context-Biasing for Production-Efficient ASR Systems](https://mlsanity.com/paper/2608.21343/) (2026)

## Abstract (arXiv)

Transducer models have emerged as a promising choice for end-to-end ASR systems, offering a balanced trade-off between recognition accuracy, streaming capabilities, and inference speed in greedy decoding. However, beam search significantly slows down Transducers due to repeated evaluations of key network components, limiting practical applications. This paper introduces a universal method to accelerate beam search for Transducers, enabling the implementation of two optimized algorithms: ALSD++ and AES++. The proposed method utilizes batch operations, a tree-based hypothesis structure, novel blank scoring for enhanced shallow fusion, and CUDA graph execution for efficient GPU inference. This narrows the speed gap between beam and greedy modes to only 10-20% for the whole system, achieves 14-30% relative improvement in WER compared to greedy decoding, and improves shallow fusion for low-resource up to 11% compared to existing implementations. All the algorithms are open sourced.