mlsanityData to 14 Oct 2026

A Review of Speaker Diarization: Recent Advances with Deep Learning

arXiv 2101.09624 ↗First posted 24 Jan 2021eess.AS cs.CL cs.SDcited by 124

Not on the map. Why?

Abstract (arXiv)

Speaker diarization is a task to label audio or video recordings with classes that correspond to speaker identity, or in short, a task to identify "who spoke when". In the early years, speaker diarization algorithms were developed for speech recognition on multispeaker audio recordings to enable speaker adaptive processing. These algorithms also gained their own value as a standalone application over time to provide speaker-specific metainformation for downstream tasks such as audio retrieval. More recently, with the emergence of deep learning technology, which has driven revolutionary changes in research and practices across speech application domains, rapid advancements have been made for speaker diarization. In this paper, we review not only the historical development of speaker diarization technology but also the recent advancements in neural speaker diarization approaches. Furthermore, we discuss how speaker diarization systems have been integrated with speech recognition applications and how the recent surge of deep learning is leading the way of jointly modeling these two components to be complementary to each other. By considering such exciting technical trends, we believe that this paper is a valuable contribution to the community to provide a survey work by consolidating the recent developments with neural methods and thus facilitating further progress toward a more efficient speaker diarization.

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Builds on 68 of its 257 references that are in the map. First posted on arXiv 24 Jan 2021.

Builds on

68 of 257 references.

References · 68

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2017Mask R-CNNcited by 7,277
  4. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
  5. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  6. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 216
  7. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  8. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  9. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 98
  10. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 92
  11. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 88
  12. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
38 more
  1. 2020Spot the conversation: speaker diarisation in the wildcited by 79
  2. 2019End-to-End Neural Speaker Diarization with Self-attentioncited by 77
  3. 2020Continuous speech separation: dataset and analysiscited by 74
  4. 2017Speaker Diarization with LSTMcited by 73
  5. 2020Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard taskscited by 73
  6. 2020End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractorscited by 71
  7. 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 68
  8. 2018Relational recurrent neural networkscited by 65
  9. 2016Achieving Human Parity in Conversational Speech Recognitioncited by 63
  10. 2016Learning Deep Embeddings with Histogram Losscited by 52
  11. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  12. 2018Fully Supervised Speaker Diarizationcited by 47
  13. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  14. 2017English Conversational Telephone Speech Recognition by Humans and Machinescited by 40
  15. 2019The Second DIHARD Diarization Challenge: Dataset, task, and baselinescited by 39
  16. 2020Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengapcited by 36
  17. 2020Integrating end-to-end neural and clustering-based diarization: Getting the best of both worldscited by 35
  18. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 35
  19. 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 33
  20. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  21. 2020Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordingscited by 28
  22. 2019Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detectioncited by 27
  23. 2020VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challengecited by 27
  24. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  25. 2018Recognizing Overlapped Speech in Meetings: A Multichannel Separation Approach Using Neural Networkscited by 23
  26. 2019Advances in Online Audio-Visual Meeting Transcriptioncited by 23
  27. 2020Speech Recognition and Multi-Speaker Diarization of Long Conversationscited by 23
  28. 2020DOVER-Lap: A Method for Combining Overlap-aware Diarization Outputscited by 19
  29. 2017Speech recognition for medical conversationscited by 18
  30. 2020Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020cited by 18
  31. 2019LSTM based Similarity Measurement with Spectral Clustering for Speaker Diarizationcited by 17
  32. 2019Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognitioncited by 16
  33. 2020Speaker Diarization with Region Proposal Networkcited by 16
  34. 2019VoxSRC 2019: The first VoxCeleb Speaker Recognition Challengecited by 14
  35. 2020Speaker Diarization with Lexical Informationcited by 14
  36. 2020Neural Speaker Diarization with Speaker-Wise Chain Rulecited by 14
  37. 2020Online End-to-End Neural Diarization with Speaker-Tracing Buffercited by 14
  38. 2019Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Modelscited by 13

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 20; 2023: 30; 2024: 29; 2025: 15; 2026: 16.

124 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2021A Survey of Sound Source Localization with Deep Learning Methodscited by 46
  4. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  5. 2021Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detectioncited by 32
  6. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  7. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  8. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  9. 2024DiarizationLM: Speaker Diarization Post-Processing with Large Language Modelscited by 16
  10. 2025SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Modelscited by 16
  11. 2021Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractorscited by 15
  12. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
38 more
  1. 2023CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videoscited by 14
  2. 2023Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correctioncited by 14
  3. 2023One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognitioncited by 12
  4. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audiocited by 10
  5. 2022Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clusteringcited by 10
  6. 2023The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systemscited by 9
  7. 2024The VoxCeleb Speaker Recognition Challenge: A Retrospectivecited by 9
  8. 2022Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractorscited by 8
  9. 2024LLM-based speaker diarization correction: A generalizable approachcited by 8
  10. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  11. 2022Using Active Speaker Faces for Diarization in TV showscited by 7
  12. 2022Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarizationcited by 7
  13. 2022Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarizationcited by 7
  14. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  15. 2024ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarizationcited by 6
  16. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  17. 2021DIVE: End-to-end Speech Diarization via Iterative Speaker Embeddingcited by 5
  18. 2022A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetingscited by 5
  19. 2022End-to-End and Self-Supervised Learning for ComParE 2022 Stuttering Sub-Challengecited by 5
  20. 2022In search of strong embedding extractors for speaker diarisationcited by 5
  21. 2023Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarizationcited by 5
  22. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  23. 2025SPGISpeech 2.0: Transcribed multi-speaker financial audio for speaker-tagged transcriptioncited by 5
  24. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 5
  25. 2021Adapting Speaker Embeddings for Speaker Diarisationcited by 4
  26. 2022Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separationcited by 4
  27. 2023End-to-End Joint Target and Non-Target Speakers ASRcited by 4
  28. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 4
  29. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 4
  30. 2021Multi-Channel End-to-End Neural Diarization with Distributed Microphonescited by 3
  31. 2022Royalflush Speaker Diarization System for ICASSP 2022 Multi-channel Multi-party Meeting Transcription Challengecited by 3
  32. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 3
  33. 2022The Conversational Short-phrase Speaker Diarization (CSSD) Task: Dataset, Evaluation Metric and Baselinescited by 3
  34. 2022BER: Balanced Error Rate For Speaker Diarizationcited by 3
  35. 2023Neural Diarization with Non-autoregressive Intermediate Attractorscited by 3
  36. 2023Diffusion-based Signal Refiner for Speech Enhancement and Separationcited by 3
  37. 2023DiaCorrect: Error Correction Back-end For Speaker Diarizationcited by 3
  38. 2023Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding with Sequence-to-Sequence Architecturecited by 3

50 most cited of 124 All 124 (JSON)