mlsanityData to 7 Oct 2026

Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator

arXiv 2305.16263 ↗First posted 25 May 2023cs.SD cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Multi-talker overlapped speech poses a significant challenge for speech recognition and diarization. Recent research indicated that these two tasks are inter-dependent and complementary, motivating us to explore a unified modeling method to address them in the context of overlapped speech. A recent study proposed a cost-effective method to convert a single-talker automatic speech recognition (ASR) system into a multi-talker one, by inserting a Sidecar separator into the frozen well-trained ASR model. Extending on this, we incorporate a diarization branch into the Sidecar, allowing for unified modeling of both ASR and diarization with a negligible overhead of only 768 parameters. The proposed method yields better ASR results compared to the baseline on LibriMix and LibriSpeechMix datasets. Moreover, without sophisticated customization on the diarization task, our method achieves acceptable diarization results on the two-speaker subset of CALLHOME with only a few adaptation steps.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

26 of 34 references.

References · 26

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  4. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  5. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  6. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  7. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  8. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  9. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  10. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  11. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  12. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 86
14 more
  1. 2019End-to-End Neural Speaker Diarization with Self-attentioncited by 74
  2. 2020End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractorscited by 70
  3. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  4. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  5. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  6. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  7. 2020Speech Recognition and Multi-Speaker Diarization of Long Conversationscited by 23
  8. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  9. 2020Speaker Diarization with Region Proposal Networkcited by 16
  10. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  11. 2019Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Modelscited by 13
  12. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  13. 2021Streaming Multi-talker Speech Recognition with Joint Speaker Identificationcited by 12
  14. 2022The CUHK-TENCENT speaker diarization system for the ICASSP 2022 multi-channel multi-party meeting transcription challengecited by 2

Built on by

Cited by year

Citing papers per year: 2024: 6; 2025: 1.

7 papers.

  1. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  2. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  3. 2024Cross-Speaker Encoding Network for Multi-Talker Speech Recognitioncited by 9
  4. 2024LLM-based speaker diarization correction: A generalizable approachcited by 8
  5. 2024Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTCcited by 4
  6. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 3
  7. 2025Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognitioncited by 2