mlsanityData to 7 Oct 2026

Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System

arXiv 2407.09817 ↗First posted 13 Jul 2024cs.SD cs.CL eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

Multi-talker speech recognition and target-talker speech recognition, both involve transcription in multi-talker contexts, remain significant challenges. However, existing methods rarely attempt to simultaneously address both tasks. In this study, we propose a pioneering approach to empower Whisper, which is a speech foundation model, to tackle joint multi-talker and target-talker speech recognition tasks. Specifically, (i) we freeze Whisper and plug a Sidecar separator into its encoder to separate mixed embedding for multiple talkers; (ii) a Target Talker Identifier is introduced to identify the embedding flow of the target talker on the fly, requiring only three-second enrollment speech as a cue; (iii) soft prompt tuning for decoder is explored for better task adaptation. Our method outperforms previous methods on two- and three-talker LibriMix and LibriSpeechMix datasets for both tasks, and delivers acceptable zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

30 of 35 references.

References · 30

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  7. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  8. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  9. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  10. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  11. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  12. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
18 more
  1. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  2. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  3. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  4. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  5. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  6. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  7. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  8. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  9. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  10. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  11. 2023Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audiocited by 10
  12. 2024Cross-Speaker Encoding Network for Multi-Talker Speech Recognitioncited by 9
  13. 2023SURT 2.0: Advances in Transducer-based Multi-talker Speech Recognitioncited by 8
  14. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  15. 2023Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speechcited by 6
  16. 2023Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognitioncited by 5
  17. 2023End-to-End Joint Target and Non-Target Speakers ASRcited by 4
  18. 2023Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  2. 2024Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTCcited by 4
  3. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  4. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 4
  5. 2025Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognitioncited by 3
  6. 2025PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systemscited by 2
  7. 2025Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognitioncited by 2
  8. 2025GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASRcited by 1
  9. 2026Xiaomi-CocktailASR-1 Technical Reportcited by 1
  10. 2026CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASRcited by 0
  11. 2026Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correctioncited by 0