# E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASR

arXiv: https://arxiv.org/abs/2204.10749 · HTML: https://mlsanity.com/paper/2204.10749/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [A comparison of end-to-end models for long-form speech recognition](https://mlsanity.com/paper/1911.02242/) (2019)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions](https://mlsanity.com/paper/2005.03271/) (2020)
- [Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging](https://mlsanity.com/paper/2012.06749/) (2020)
- [End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection](https://mlsanity.com/paper/2002.00551/) (2020)
- [Endpoint Detection for Streaming End-to-End Multi-talker ASR](https://mlsanity.com/paper/2201.09979/) (2022)
- [Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models](https://mlsanity.com/paper/2106.06126/) (2021)
- [VADOI:Voice-Activity-Detection Overlapping Inference For End-to-end Long-form Speech Recognition](https://mlsanity.com/paper/2202.10593/) (2022)

## Built on by (most cited first, 17 of 17)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study](https://mlsanity.com/paper/2401.12789/) (2024)
- [Updated Corpora and Benchmarks for Long-Form Speech Recognition](https://mlsanity.com/paper/2309.15013/) (2023)
- [InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model](https://mlsanity.com/paper/2503.02969/) (2025)
- [Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio](https://mlsanity.com/paper/2505.10975/) (2025)
- [Smart Speech Segmentation using Acousto-Linguistic Features with look-ahead](https://mlsanity.com/paper/2210.14446/) (2022)
- [E2E Segmentation in a Two-Pass Cascaded Encoder ASR Model](https://mlsanity.com/paper/2211.15432/) (2022)
- [Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation](https://mlsanity.com/paper/2306.01296/) (2023)
- [Long-form Simultaneous Speech Translation: Thesis Proposal](https://mlsanity.com/paper/2310.11141/) (2023)
- [Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding](https://mlsanity.com/paper/2603.06193/) (2026)
- [Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition](https://mlsanity.com/paper/2301.03819/) (2023)
- [Semantic Segmentation with Bidirectional Language Models Improves Long-form ASR](https://mlsanity.com/paper/2305.18419/) (2023)
- [Adaptive Endpointing with Deep Contextual Multi-armed Bandits](https://mlsanity.com/paper/2303.13407/) (2023)
- [Text Injection for Capitalization and Turn-Taking Prediction in Speech Models](https://mlsanity.com/paper/2308.07395/) (2023)
- [Memory-augmented conformer for improved end-to-end long-form ASR](https://mlsanity.com/paper/2309.13029/) (2023)
- [Segmental Attention Decoding With Long Form Acoustic Encodings](https://mlsanity.com/paper/2512.14652/) (2025)
- [Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents](https://mlsanity.com/paper/2608.22101/) (2026)

## Abstract (arXiv)

Improving the performance of end-to-end ASR models on long utterances ranging from minutes to hours in length is an ongoing challenge in speech recognition. A common solution is to segment the audio in advance using a separate voice activity detector (VAD) that decides segment boundary locations based purely on acoustic speech/non-speech information. VAD segmenters, however, may be sub-optimal for real-world speech where, e.g., a complete sentence that should be taken as a whole may contain hesitations in the middle ("set an alarm for... 5 o'clock"). We propose to replace the VAD with an end-to-end ASR model capable of predicting segment boundaries in a streaming fashion, allowing the segmentation decision to be conditioned not only on better acoustic features but also on semantic features from the decoded text with negligible extra computation. In experiments on real world long-form audio (YouTube) with lengths of up to 30 minutes, we demonstrate 8.5% relative WER improvement and 250 ms reduction in median end-of-segment latency compared to the VAD segmenter baseline on a state-of-the-art Conformer RNN-T model.