# Reducing Streaming ASR Model Delay with Self Alignment

arXiv: https://arxiv.org/abs/2105.05005 · HTML: https://mlsanity.com/paper/2105.05005/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition](https://mlsanity.com/paper/2010.03192/) (2020)
- [Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR](https://mlsanity.com/paper/2004.05009/) (2020)

## Built on by (most cited first, 16 of 16)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks](https://mlsanity.com/paper/2210.07499/) (2022)
- [Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2211.02333/) (2022)
- [Have best of both worlds: two-pass hybrid and E2E cascading framework for speech recognition](https://mlsanity.com/paper/2110.04891/) (2021)
- [Extracting Targeted Training Data from ASR Models, and How to Mitigate It](https://mlsanity.com/paper/2204.08345/) (2022)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [Powerful and Extensible WFST Framework for RNN-Transducer Losses](https://mlsanity.com/paper/2303.10384/) (2023)
- [Federated Domain Adaptation for ASR with Full Self-Supervision](https://mlsanity.com/paper/2203.15966/) (2022)
- [CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition](https://mlsanity.com/paper/2307.14132/) (2023)
- [Unified Segment-to-Segment Framework for Simultaneous Sequence Generation](https://mlsanity.com/paper/2310.17940/) (2023)
- [StableEmit: Selection Probability Discount for Reducing Emission Latency of Streaming Monotonic Attention ASR](https://mlsanity.com/paper/2107.00635/) (2021)
- [Bayes Risk Transducer: Transducer with Controllable Alignment Prediction](https://mlsanity.com/paper/2308.10107/) (2023)
- [MAC: A unified framework boosting low resource automatic speech recognition](https://mlsanity.com/paper/2302.03498/) (2023)
- [A Differentiable Alignment Framework for Sequence-to-Sequence Modeling via Optimal Transport](https://mlsanity.com/paper/2502.01588/) (2025)
- [Improving endpoint detection in end-to-end streaming ASR for conversational speech](https://mlsanity.com/paper/2505.17070/) (2025)

## Abstract (arXiv)

Reducing prediction delay for streaming end-to-end ASR models with minimal performance regression is a challenging problem. Constrained alignment is a well-known existing approach that penalizes predicted word boundaries using external low-latency acoustic models. On the contrary, recently proposed FastEmit is a sequence-level delay regularization scheme encouraging vocabulary tokens over blanks without any reference alignments. Although all these schemes are successful in reducing delay, ASR word error rate (WER) often severely degrades after applying these delay constraining schemes. In this paper, we propose a novel delay constraining method, named self alignment. Self alignment does not require external alignment models. Instead, it utilizes Viterbi forced-alignments from the trained model to find the lower latency alignment direction. From LibriSpeech evaluation, self alignment outperformed existing schemes: 25% and 56% less delay compared to FastEmit and constrained alignment at the similar word error rate. For Voice Search evaluation,12% and 25% delay reductions were achieved compared to FastEmit and constrained alignment with more than 2% WER improvements.