mlsanityData to 7 Oct 2026

A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio

arXiv 2107.02852 ↗First posted 6 Jul 2021eess.AS cs.CL cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

Speaker-attributed automatic speech recognition (SA-ASR) is a task to recognize "who spoke what" from multi-talker recordings. An SA-ASR system usually consists of multiple modules such as speech separation, speaker diarization and ASR. On the other hand, considering the joint optimization, an end-to-end (E2E) SA-ASR model has recently been proposed with promising results on simulation data. In this paper, we present our recent study on the comparison of such modular and joint approaches towards SA-ASR on real monaural recordings. We develop state-of-the-art SA-ASR systems for both modular and joint approaches by leveraging large-scale training data, including 75 thousand hours of ASR training data and the VoxCeleb corpus for speaker representation learning. We also propose a new pipeline that performs the E2E SA-ASR model after speaker clustering. Our evaluation on the AMI meeting corpus reveals that after fine-tuning with a small real data, the joint system performs 8.9--29.9% better in accuracy compared to the best modular system while the modular system performs better before such fine-tuning. We also conduct various error analyses to show the remaining issues for the monaural SA-ASR.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

38 of 57 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  7. 2019Res2Net: A New Multi-scale Backbone Architecturecited by 468
  8. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  9. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  10. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  11. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  12. 2020Continuous speech separation: dataset and analysiscited by 73
26 more
  1. 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 67
  2. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  3. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 44
  4. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  5. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  6. 2020Continuous Speech Separation with Conformercited by 36
  7. 2020Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengapcited by 35
  8. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  9. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 34
  10. 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 33
  11. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  12. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  13. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  14. 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 28
  15. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  16. 2020VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challengecited by 26
  17. 2019Advances in Online Audio-Visual Meeting Transcriptioncited by 23
  18. 2020Speech Recognition and Multi-Speaker Diarization of Long Conversationscited by 23
  19. 2020Streaming Multi-speaker ASR with RNN-Tcited by 22
  20. 2020Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020cited by 17
  21. 2019Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognitioncited by 16
  22. 2019Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Modelscited by 13
  23. 2020Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASRcited by 11
  24. 2019Guided Source Separation Meets a Strong ASR Backend: Hitachi/Paderborn University Joint Investigation for Dinner Party ASRcited by 10
  25. 2020Tackling real noisy reverberant meetings with all-neural source separation, counting, and diarization systemcited by 10
  26. 2021Hypothesis Stitcher for End-to-End Speaker-attributed ASR on Long-form Multi-talker Recordingscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 0; 2024: 1; 2025: 1; 2026: 1.

10 papers.

  1. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  2. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  3. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  4. 2021Multi-turn RNN-T for streaming recognition of multi-party speechcited by 10
  5. 2022A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetingscited by 5
  6. 2024MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Modelscited by 4
  7. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  8. 2022The RoyalFlush System of Speech Recognition for M2MeT Challengecited by 3
  9. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 3
  10. 2026HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Groundingcited by 0