# Endpoint Detection for Streaming End-to-End Multi-talker ASR

arXiv: https://arxiv.org/abs/2201.09979 · HTML: https://mlsanity.com/paper/2201.09979/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2022.

## Built on by (most cited first, 10 of 10)

- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications](https://mlsanity.com/paper/2203.16822/) (2022)
- [E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASR](https://mlsanity.com/paper/2204.10749/) (2022)
- [Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems](https://mlsanity.com/paper/2211.00786/) (2022)
- [Alignment-Free Training for Transducer-based Multi-Talker ASR](https://mlsanity.com/paper/2409.20301/) (2024)
- [Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speech](https://mlsanity.com/paper/2205.05199/) (2022)
- [Smart Speech Segmentation using Acousto-Linguistic Features with look-ahead](https://mlsanity.com/paper/2210.14446/) (2022)
- [Adaptive Endpointing with Deep Contextual Multi-armed Bandits](https://mlsanity.com/paper/2303.13407/) (2023)
- [Improving endpoint detection in end-to-end streaming ASR for conversational speech](https://mlsanity.com/paper/2505.17070/) (2025)
- [A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI](https://mlsanity.com/paper/2603.13379/) (2026)

## Abstract (arXiv)

Streaming end-to-end multi-talker speech recognition aims at transcribing the overlapped speech from conversations or meetings with an all-neural model in a streaming fashion, which is fundamentally different from a modular-based approach that usually cascades the speech separation and the speech recognition models trained independently. Previously, we proposed the Streaming Unmixing and Recognition Transducer (SURT) model based on recurrent neural network transducer (RNN-T) for this problem and presented promising results. However, for real applications, the speech recognition system is also required to determine the timestamp when a speaker finishes speaking for prompt system response. This problem, known as endpoint (EP) detection, has not been studied previously for multi-talker end-to-end models. In this work, we address the EP detection problem in the SURT framework by introducing an end-of-sentence token as an output unit, following the practice of single-talker end-to-end models. Furthermore, we also present a latency penalty approach that can significantly cut down the EP detection latency. Our experimental results based on the 2-speaker LibrispeechMix dataset show that the SURT model can achieve promising EP detection without significantly degradation of the recognition accuracy.