mlsanityData to 14 Oct 2026

Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition

arXiv 2510.03723 ↗First posted 4 Oct 2025eess.AS cs.CL cs.SDcited by 2

Not on the map. Why?

Abstract (arXiv)

We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix).

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

26 of 33 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2015Listen, Attend and Spellcited by 543
  5. 2015Attention-Based Models for Speech Recognitioncited by 401
  6. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 191
  7. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  8. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  9. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  10. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  11. 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
  12. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
14 more
  1. 2023MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systemscited by 28
  2. 2024Leveraging Self-Supervised Learning for Speaker Diarizationcited by 23
  3. 2024NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcriptioncited by 21
  4. 2024Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systemscited by 15
  5. 2023One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognitioncited by 12
  6. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  7. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  8. 2024The CHiME-8 DASR Challenge for Generalizable and Array Agnostic Distant Automatic Speech Recognition and Diarizationcited by 10
  9. 2024Target Speaker ASR with Whispercited by 9
  10. 2023SURT 2.0: Advances in Transducer-based Multi-talker Speech Recognitioncited by 8
  11. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  12. 2025DNCASR: End-to-End Training for Speaker-Attributed ASRcited by 5
  13. 2024META-CAT: Speaker-Informed Speech Embeddings via Meta Information Concatenation for Multi-talker ASRcited by 4
  14. 2025Summary of the NOTSOFAR-1 Challenge: Highlights and Learningscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whispercited by 2
  2. 2026Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetingscited by 0