mlsanityData to 7 Oct 2026

One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition

arXiv 2310.01688 ↗First posted 2 Oct 2023eess.AS cs.CL cs.SDcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel framework for joint speaker diarization (SD) and automatic speech recognition (ASR), named SLIDAR (sliding-window diarization-augmented recognition). SLIDAR can process arbitrary length inputs and can handle any number of speakers, effectively solving ``who spoke what, when'' concurrently. SLIDAR leverages a sliding window approach and consists of an end-to-end diarization-augmented speech transcription (E2E DAST) model which provides, locally, for each window: transcripts, diarization and speaker embeddings. The E2E DAST model is based on an encoder-decoder architecture and leverages recent techniques such as serialized output training and ``Whisper-style" prompting. The local outputs are then combined to get the final SD+ASR result by clustering the speaker embeddings to get global speaker identities. Experiments performed on monaural recordings from the AMI corpus confirm the effectiveness of the method in both close-talk and far-field speech scenarios.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

38 of 51 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  6. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 131
  7. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  8. 2023End-to-End Speech Recognition: A Surveycited by 127
  9. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  10. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  11. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  12. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
26 more
  1. 2020Continuous speech separation: dataset and analysiscited by 73
  2. 2020Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard taskscited by 69
  3. 2021M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challengecited by 67
  4. 2019SpecAugment on Large Scale Datasetscited by 60
  5. 2020End-to-End Multi-speaker Speech Recognition with Transformercited by 42
  6. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  7. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  8. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 34
  9. 2020Integrating end-to-end neural and clustering-based diarization: Getting the best of both worldscited by 33
  10. 2020Streaming end-to-end multi-talker speech recognitioncited by 29
  11. 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 28
  12. 2023MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systemscited by 28
  13. 2021Advances in integration of end-to-end neural and clustering-based diarization for real conversational speechcited by 22
  14. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  15. 2023The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarioscited by 20
  16. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  17. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 17
  18. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  19. 2023TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddingscited by 12
  20. 2022Summary On The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Grand Challengecited by 11
  21. 2021Continuous Streaming Multi-Talker ASR with Dual-path Transducerscited by 10
  22. 2021VarArray: Array-Geometry-Agnostic Continuous Speech Separationcited by 10
  23. 2022EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakerscited by 10
  24. 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
  25. 2020Multi-class Spectral Clustering with Overlaps for Speaker Diarizationcited by 6
  26. 2022Low-Latency Speech Separation Guided Diarization for Telephone Conversationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 2.

12 papers.

  1. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  2. 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
  3. 2024The VoxCeleb Speaker Recognition Challenge: A Retrospectivecited by 9
  4. 2024LLM-based speaker diarization correction: A generalizable approachcited by 8
  5. 2024MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Modelscited by 4
  6. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  7. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 4
  8. 2025A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversationscited by 2
  9. 2025Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognitioncited by 2
  10. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whispercited by 2
  11. 2024Joint vs Sequential Speaker-Role Detection and Automatic Speech Recognition for Air-traffic Controlcited by 1
  12. 2026Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metricscited by 0