# Improving endpoint detection in end-to-end streaming ASR for conversational speech

arXiv: https://arxiv.org/abs/2505.17070 · HTML: https://mlsanity.com/paper/2505.17070/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 16 of 16)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Online and Linear-Time Attention by Enforcing Monotonic Alignments](https://mlsanity.com/paper/1704.00784/) (2017)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion](https://mlsanity.com/paper/2401.14717/) (2024)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Reducing Streaming ASR Model Delay with Self Alignment](https://mlsanity.com/paper/2105.05005/) (2021)
- [Endpoint Detection for Streaming End-to-End Multi-talker ASR](https://mlsanity.com/paper/2201.09979/) (2022)
- [Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2211.02333/) (2022)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems](https://mlsanity.com/paper/2211.00786/) (2022)
- [A comparison of streaming models and data augmentation methods for robust speech recognition](https://mlsanity.com/paper/2111.10043/) (2021)

## Abstract (arXiv)

ASR endpointing (EP) plays a major role in delivering a good user experience in products supporting human or artificial agents in human-human/machine conversations. Transducer-based ASR (T-ASR) is an end-to-end (E2E) ASR modelling technique preferred for streaming. A major limitation of T-ASR is delayed emission of ASR outputs, which could lead to errors or delays in EP. Inaccurate EP will cut the user off while speaking, returning incomplete transcript while delays in EP will increase the perceived latency, degrading the user experience. We propose methods to improve EP by addressing delayed emission along with EP mistakes. To address the delayed emission problem, we introduce an end-of-word token at the end of each word, along with a delay penalty. The EP delay is addressed by obtaining a reliable frame-level speech activity detection using an auxiliary network. We apply the proposed methods on Switchboard conversational speech corpus and evaluate it against a delay penalty method.