# Factorized Blank Thresholding for Improved Runtime Efficiency of Neural Transducers

arXiv: https://arxiv.org/abs/2211.00896 · HTML: https://mlsanity.com/paper/2211.00896/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2022.

## Built on by (most cited first, 6 of 6)

- [Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition](https://mlsanity.com/paper/2309.07988/) (2023)
- [Breaking Down Power Barriers in On-Device Streaming ASR: Insights and Solutions](https://mlsanity.com/paper/2402.13076/) (2024)
- [A Token-Wise Beam Search Algorithm for RNN-T](https://mlsanity.com/paper/2302.14357/) (2023)
- [Improved Factorized Neural Transducer Model For text-only Domain Adaptation](https://mlsanity.com/paper/2309.09524/) (2023)
- [Boosting Hybrid Autoregressive Transducer-based ASR with Internal Acoustic Model Training and Dual Blank Thresholding](https://mlsanity.com/paper/2409.20313/) (2024)
- [Factorized Delayed Streams Modeling for LLM-based Streaming ASR](https://mlsanity.com/paper/2610.04333/) (2026)

## Abstract (arXiv)

We show how factoring the RNN-T's output distribution can significantly reduce the computation cost and power consumption for on-device ASR inference with no loss in accuracy. With the rise in popularity of neural-transducer type models like the RNN-T for on-device ASR, optimizing RNN-T's runtime efficiency is of great interest. While previous work has primarily focused on the optimization of RNN-T's acoustic encoder and predictor, this paper focuses the attention on the joiner. We show that despite being only a small part of RNN-T, the joiner has a large impact on the overall model's runtime efficiency. We propose to utilize HAT-style joiner factorization for the purpose of skipping the more expensive non-blank computation when the blank probability exceeds a certain threshold. Since the blank probability can be computed very efficiently and the RNN-T output is dominated by blanks, our proposed method leads to a 26-30% decoding speed-up and 43-53% reduction in on-device power consumption, all the while incurring no accuracy degradation and being relatively simple to implement.