mlsanityData to 19 Oct 2026

Pushing the Boundaries of Streaming Multi-Speaker ASR: A Systematic Study of Architectural Trade-offs

arXiv 2609.10265 ↗First posted 9 Sep 2026eess.AS cs.SDcited by 0

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

Streaming multi-speaker ASR is a challenging task that must balance accuracy, latency, and efficiency while handling overlapping speech and maintaining coherent long-context modeling over extended conversations in an online fashion. We present a unified framework that categorizes streaming multi-speaker ASR into four architectural strategies based on how diarization and ASR are integrated. Using a shared pair of open-source streaming ASR and diarization models as a common foundation, we derive four multi-speaker ASR systems that differ in whether they employ multiple model instances, fine-tuning, or both. We evaluate these systems across multi-speaker accuracy, single-speaker accuracy degradation, memory footprint, and training complexity. Through this systematic architectural analysis, we clarify the design space for streaming multi-speaker ASR and provide practical guidance for selecting the most suitable approach under diverse deployment constraints.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 30 of its 42 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

30 of 42 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper27 Jan
  2. 2025Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering24 Jul
  3. 2025Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR27 Jun
  4. 2025Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios17 Jun
  5. 2024Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems10 Sep
  6. 2024Serialized Output Training by Learned Dominance4 Jul
  7. 2024SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR4 Mar
  8. 2024NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription16 Jan
  9. 2023One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition2 Oct
  10. 2023Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approach11 Sep
  11. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings30 Mar
  12. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Training2 Feb
1 more
  1. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone31 Mar

Other references · 17

  1. 2017Attention Is All You Needcited by 61,180
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 545
  4. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 231
  5. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 170
  6. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 144
  7. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 131
  8. 2020Continuous speech separation: dataset and analysiscited by 124
  9. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 67
  10. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 66
  11. 2019DiPCo -- Dinner Party Corpuscited by 42
  12. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 38
5 more
  1. 2023Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognitioncited by 36
  2. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 32
  3. 2025Granary: Speech Recognition and Translation Dataset in 25 European Languagescited by 31
  4. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 31
  5. 2026VIBEVOICE-ASR Technical Reportcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FASTDIAR: Frame-level speaker encoder for Streaming Diarization
  2. 2025Unifying Streaming and Non-streaming Zipformer-based ASR
  3. 2025Scaling Multi-Talker ASR with Speaker-Agnostic Activity Streams
  4. 2026DM-ASR: Diarization-aware Multi-speaker ASR with Large Language Models
  5. 2021Continuous Streaming Multi-Talker ASR with Dual-path Transducers
  6. 2024Target Speaker ASR with Whisper
  7. 2026VibeVoice-ASR-Streaming Technical Report
  8. 2026X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR