mlsanityData to 14 Oct 2026

Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder

arXiv 2508.20474 ↗First posted 28 Aug 2025eess.AS cs.CL cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks using a shared speech foundational encoder. We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data. Our evaluations demonstrate that UME substantially improves over the single-task baselines dedicated to SD, SS, and multi-speaker ASR on LibriMix evaluation sets. Notably, for SD, UME outperforms the previous studies, achieving diarization error rates of 1.37% and 2.29% on Libri2Mix and Libri3Mix evaluation sets, respectively.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 28 Aug 2025.

Builds on

43 of 50 references.

References · 43

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  7. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  8. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 216
  9. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  10. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  11. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 124
  12. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 111
31 more
  1. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  2. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  3. 2020Continuous speech separation: dataset and analysiscited by 74
  4. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  5. 2022TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separationcited by 48
  6. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 44
  7. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  8. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 37
  9. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 35
  10. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  11. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  12. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  13. 2020Speech Recognition and Multi-Speaker Diarization of Long Conversationscited by 23
  14. 2021Encoder-Decoder Based Attractors for End-to-End Neural Diarizationcited by 20
  15. 2017Single-Channel Multi-talker Speech Recognition with Permutation Invariant Trainingcited by 16
  16. 2020Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASRcited by 15
  17. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  18. 2023TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddingscited by 12
  19. 2023One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognitioncited by 12
  20. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  21. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  22. 2019End-to-end training of time domain audio separation and recognitioncited by 8
  23. 2021The Hitachi-JHU DIHARD III System: Competitive End-to-End Neural Diarization and X-Vector Clustering Systems Combined by DOVER-Lapcited by 8
  24. 2024PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordingscited by 8
  25. 2022Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcriptioncited by 6
  26. 2024Neural Blind Source Separation and Diarization for Distant Speech Recognitioncited by 6
  27. 2024The USTC-NERCSLIP Systems for the CHiME-8 NOTSOFAR-1 Challengecited by 5
  28. 2024NTT Multi-Speaker ASR System for the DASR Task of CHiME-8 Challengecited by 5
  29. 2019The Power of the Weighted Sum Scalarization for Approximating Multiobjective Optimization Problemscited by 4
  30. 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
  31. 2024STCON System for the CHiME-8 Challengecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Groundingcited by 5
  2. 2026CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASRcited by 0
  3. 2026Modeling Overlapped Speech with Shufflescited by 0
  4. 2026Multi-task Learning is Not Enough: Representational Entanglement in Dual-output Second Language Speech Recognitioncited by 0
  5. 2026SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separationcited by 0