# Delay-penalized transducer for low-latency streaming ASR

arXiv: https://arxiv.org/abs/2211.00490 · HTML: https://mlsanity.com/paper/2211.00490/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2012.05481/) (2020)
- [Lhotse: a speech data representation library for the modern deep learning ecosystem](https://mlsanity.com/paper/2110.12561/) (2021)
- [Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition](https://mlsanity.com/paper/1507.06947/) (2015)
- [Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR](https://mlsanity.com/paper/2004.05009/) (2020)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Reducing Streaming ASR Model Delay with Self Alignment](https://mlsanity.com/paper/2105.05005/) (2021)

## Built on by (most cited first, 7 of 7)

- [Mamba for Streaming ASR Combined with Unimodal Aggregation](https://mlsanity.com/paper/2410.00070/) (2024)
- [Bayes Risk Transducer: Transducer with Controllable Alignment Prediction](https://mlsanity.com/paper/2308.10107/) (2023)
- [Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASR](https://mlsanity.com/paper/2505.22069/) (2025)
- [A Differentiable Alignment Framework for Sequence-to-Sequence Modeling via Optimal Transport](https://mlsanity.com/paper/2502.01588/) (2025)
- [Improving endpoint detection in end-to-end streaming ASR for conversational speech](https://mlsanity.com/paper/2505.17070/) (2025)
- [Do we really need Self-Attention for Streaming Automatic Speech Recognition?](https://mlsanity.com/paper/2601.19960/) (2026)
- [SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition](https://mlsanity.com/paper/2603.10005/) (2026)

## Abstract (arXiv)

In streaming automatic speech recognition (ASR), it is desirable to reduce latency as much as possible while having minimum impact on recognition accuracy. Although a few existing methods are able to achieve this goal, they are difficult to implement due to their dependency on external alignments. In this paper, we propose a simple way to penalize symbol delay in transducer model, so that we can balance the trade-off between symbol delay and accuracy for streaming models without external alignments. Specifically, our method adds a small constant times (T/2 - t), where T is the number of frames and t is the current frame, to all the non-blank log-probabilities (after normalization) that are fed into the two dimensional transducer recursion. For both streaming Conformer models and unidirectional long short-term memory (LSTM) models, experimental results show that it can significantly reduce the symbol delay with an acceptable performance degradation. Our method achieves similar delay-accuracy trade-off to the previously published FastEmit, but we believe our method is preferable because it has a better justification: it is equivalent to penalizing the average symbol delay. Our work is open-sourced and publicly available (https://github.com/k2-fsa/k2).