mlsanityData to 14 Oct 2026

Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio

arXiv 2505.10975 ↗First posted 16 May 2025cs.CL cs.AI cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Monaural multi-speaker automatic speech recognition (ASR) remains challenging due to data scarcity and the intrinsic difficulty of recognizing and attributing words to individual speakers, particularly in overlapping speech. Recent advances have driven the shift from cascade systems to end-to-end (E2E) architectures, which reduce error propagation and better exploit the synergy between speech content and speaker identity. Despite rapid progress in E2E multi-speaker ASR, the field lacks a comprehensive review of recent developments. This survey provides a systematic taxonomy of E2E neural approaches for multi-speaker ASR, highlighting recent advances and comparative analysis. Specifically, we analyze: (1) architectural paradigms (SIMO vs.~SISO) for pre-segmented audio, analyzing their distinct characteristics and trade-offs; (2) recent architectural and algorithmic improvements based on these two paradigms; (3) extensions to long-form speech, including segmentation strategy and speaker-consistent hypothesis stitching. Further, we (4) evaluate and compare methods across standard benchmarks. We conclude with a discussion of open challenges and future research directions towards building robust and scalable multi-speaker ASR.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 73 of its 103 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

73 of 103 references.

References · 73

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  7. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  8. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  9. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  10. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 191
  11. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  12. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 181
38 more
  1. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  2. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  3. 2023End-to-End Speech Recognition: A Surveycited by 129
  4. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  5. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  6. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 88
  7. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  8. 2020Continuous speech separation: dataset and analysiscited by 74
  9. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  10. 2020Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard taskscited by 73
  11. 2021M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challengecited by 71
  12. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  13. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  14. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  15. 2023Sparks of Large Audio Models: A Survey and Outlookcited by 41
  16. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  17. 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 33
  18. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  19. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  20. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  21. 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 28
  22. 2023MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systemscited by 28
  23. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  24. 2020Speech Recognition and Multi-Speaker Diarization of Long Conversationscited by 23
  25. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  26. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  27. 2022E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASRcited by 17
  28. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  29. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  30. 2020Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASRcited by 15
  31. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  32. 2023One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognitioncited by 12
  33. 2020Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASRcited by 11
  34. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  35. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  36. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audiocited by 10
  37. 2022Simulating realistic speech overlaps improves multi-talker ASRcited by 10
  38. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptionscited by 3
  2. 2026Attention-guided Evidence Grounding for Spoken Question Answeringcited by 0
  3. 2026Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metricscited by 0
  4. 2026H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASRcited by 0