mlsanityData to 19 Oct 2026

Elevating Robust Multi-Talker ASR by Decoupling Speaker Separation and Speech Recognition

arXiv 2503.17886 ↗First posted 23 Mar 2025cs.SD eess.AScited by 0

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

Despite the tremendous success of automatic speech recognition (ASR) with the introduction of deep learning, its performance is still unsatisfactory in many real-world multi-talker scenarios. Speaker separation excels in separating individual talkers but, as a frontend, it introduces processing artifacts that degrade the ASR backend trained on clean speech. As a result, mainstream robust ASR systems train the backend on noisy speech to avoid processing artifacts. In this work, we propose to decouple the training of the speaker separation frontend and the ASR backend, with the latter trained on clean speech only. Our decoupled system achieves 5.1% word error rates (WER) on the Libri2Mix dev/test sets, significantly outperforming other multi-talker ASR baselines. Its effectiveness is also demonstrated with the state-of-the-art 7.60%/5.74% WERs on 1-ch and 6-ch SMS-WSJ. Furthermore, on recorded LibriCSS, we achieve the speaker-attributed WER of 2.92%. These state-of-the-art results suggest that decoupling speaker separation and recognition is an effective approach to elevate robust multi-talker ASR.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 42 of its 65 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

42 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DCF-DS: Deep Cascade Fusion of Diarization and Separation for Speech Recognition under Realistic Single-Channel Conditions11 Nov
  2. 2023Multi-channel Conversational Speaker Separation via Neural Diarization15 Nov
  3. 2023Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representation23 Jul
  4. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation22 Nov
  5. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR7 Oct
  6. 2021Localization Based Sequential Grouping for Continuous Speech Separation14 Jul

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 38,210
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 2,293
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  5. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 754
  6. 2018SDR - half-baked or well done?cited by 626
  7. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 596
  8. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 342
  9. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 275
  10. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 246
  11. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 231
  12. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 219
24 more
  1. 2023End-to-End Speech Recognition: A Surveycited by 163
  2. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 138
  3. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 131
  4. 2020Continuous speech separation: dataset and analysiscited by 124
  5. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 71
  6. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 58
  7. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 44
  8. 2020Streaming end-to-end multi-talker speech recognitioncited by 41
  9. 2023SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberationcited by 39
  10. 2019Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modelingcited by 36
  11. 2020Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separationcited by 35
  12. 2023TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddingscited by 27
  13. 2017Single-Channel Multi-talker Speech Recognition with Permutation Invariant Trainingcited by 26
  14. 2020Multi-Microphone Complex Spectral Mapping for Speech Dereverberationcited by 25
  15. 2019SMS-WSJ: Database, performance measures, and baseline recipe for multi-channel source separation and recognitioncited by 22
  16. 2021Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chaincited by 18
  17. 2020Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognitioncited by 17
  18. 2020On End-to-end Multi-channel Time Domain Speech Separation in Reverberant Environmentscited by 17
  19. 2021Self-attending RNN for Speech Enhancement to Improve Cross-corpus Generalizationcited by 14
  20. 2021Speaker activity driven neural speech extractioncited by 11
  21. 2023Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognitioncited by 9
  22. 2021Convolutive Prediction for Reverberant Speech Separationcited by 6
  23. 2024Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASRcited by 5
  24. 2022Collar-aware Training for Streaming Speaker Change Detection in Broadcast Speechcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixture Encoder for Joint Speech Separation and Recognition
  2. 2021Investigation of Practical Aspects of Single Channel Speech Separation for ASR
  3. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One
  4. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio
  5. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization
  6. 2021Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlapping Speech Recognition
  7. 2025Advances in Speech Separation: Techniques, Challenges, and Future Trends
  8. 2023Cascaded encoders for fine-tuning ASR models on overlapped speech