Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios
Not on the map. Why?
Abstract (arXiv)
We extend the frameworks of Serialized Output Training (SOT) to address practical needs of both streaming and offline automatic speech recognition (ASR) applications. Our approach focuses on balancing latency and accuracy, catering to real-time captioning and summarization requirements. We propose several key improvements: (1) Leveraging Continuous Speech Separation (CSS) single-channel front-end with end-to-end (E2E) systems for highly overlapping scenarios, challenging the conventional wisdom of E2E versus cascaded setups. The CSS framework improves the accuracy of the ASR system by separating overlapped speech from multiple speakers. (2) Implementing dual models -- Conformer Transducer for streaming and Sequence-to-Sequence for offline -- or alternatively, a two-pass model based on cascaded encoders. (3) Exploring segment-based SOT (segSOT) which is better suited for offline scenarios while also enhancing readability of multi-talker transcriptions.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 17 Jun 2025.
Builds on
28 of 33 references.
References · 28
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2015Listen, Attend and Spellcited by 543
- 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
- 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 191
- 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
- 2023End-to-End Speech Recognition: A Surveycited by 129
- 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 100
- 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
- 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
- 2020Continuous speech separation: dataset and analysiscited by 74
16 more
- 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
- 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
- 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 53
- 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 44
- 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
- 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
- 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
- 2020Continuous Speech Separation with Conformercited by 36
- 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 35
- 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 33
- 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 28
- 2020High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Modelcited by 8
- 2020Directional ASR: A New Paradigm for E2E Multi-Speaker Speech Recognition with Source Localizationcited by 8
- 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
- 2022Speech separation with large-scale self-supervised learningcited by 5
- 2022Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Expertscited by 3
Built on by
Not cited yet.