# Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR

arXiv: https://arxiv.org/abs/2004.05009 · HTML: https://mlsanity.com/paper/2004.05009/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 19 of 19)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Large-Scale Streaming End-to-End Speech Translation with Neural Transducers](https://mlsanity.com/paper/2204.05352/) (2022)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Reducing Streaming ASR Model Delay with Self Alignment](https://mlsanity.com/paper/2105.05005/) (2021)
- [Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition](https://mlsanity.com/paper/2112.05820/) (2021)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2211.02333/) (2022)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition](https://mlsanity.com/paper/2307.14132/) (2023)
- [Unified Segment-to-Segment Framework for Simultaneous Sequence Generation](https://mlsanity.com/paper/2310.17940/) (2023)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)
- [Streaming Sequence Transduction through Dynamic Compression](https://mlsanity.com/paper/2402.01172/) (2024)
- [StableEmit: Selection Probability Discount for Reducing Emission Latency of Streaming Monotonic Attention ASR](https://mlsanity.com/paper/2107.00635/) (2021)
- [VAD-free Streaming Hybrid CTC/Attention ASR for Unsegmented Recording](https://mlsanity.com/paper/2107.07509/) (2021)
- [Mutually-Constrained Monotonic Multihead Attention for Online ASR](https://mlsanity.com/paper/2103.14302/) (2021)
- [An Investigation of Enhancing CTC Model for Triggered Attention-based Streaming ASR](https://mlsanity.com/paper/2110.10402/) (2021)
- [Dynamic Latency for CTC-Based Streaming Automatic Speech Recognition With Emformer](https://mlsanity.com/paper/2203.15613/) (2022)
- [Globally Normalising the Transducer for Streaming Speech Recognition](https://mlsanity.com/paper/2307.10975/) (2023)
- [X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR](https://mlsanity.com/paper/2609.08672/) (2026)

## Abstract (arXiv)

Recently, a few novel streaming attention-based sequence-to-sequence (S2S) models have been proposed to perform online speech recognition with linear-time decoding complexity. However, in these models, the decisions to generate tokens are delayed compared to the actual acoustic boundaries since their unidirectional encoders lack future information. This leads to an inevitable latency during inference. To alleviate this issue and reduce latency, we propose several strategies during training by leveraging external hard alignments extracted from the hybrid model. We investigate to utilize the alignments in both the encoder and the decoder. On the encoder side, (1) multi-task learning and (2) pre-training with the framewise classification task are studied. On the decoder side, we (3) remove inappropriate alignment paths beyond an acceptable latency during the alignment marginalization, and (4) directly minimize the differentiable expected latency loss. Experiments on the Cortana voice search task demonstrate that our proposed methods can significantly reduce the latency, and even improve the recognition accuracy in certain cases on the decoder side. We also present some analysis to understand the behaviors of streaming S2S models.