# Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds

arXiv: https://arxiv.org/abs/2010.13366 · HTML: https://mlsanity.com/paper/2010.13366/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2020.

## Built on by (most cited first, 30 of 35)

- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2110.03151/) (2021)
- [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition](https://mlsanity.com/paper/2310.01688/) (2023)
- [Multi-scale Speaker Diarization with Dynamic Scale Weighting](https://mlsanity.com/paper/2203.15974/) (2022)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [The VoxCeleb Speaker Recognition Challenge: A Retrospective](https://mlsanity.com/paper/2408.14886/) (2024)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization](https://mlsanity.com/paper/2208.13085/) (2022)
- [Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Models](https://mlsanity.com/paper/2506.18623/) (2025)
- [DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding](https://mlsanity.com/paper/2105.13802/) (2021)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [Exploring Speech Foundation Models for Speaker Diarization in Child-Adult Dyadic Interactions](https://mlsanity.com/paper/2406.07890/) (2024)
- [Mamba-based Segmentation Model for Speaker Diarization](https://mlsanity.com/paper/2410.06459/) (2024)
- [Tight integration of neural- and clustering-based diarization through deep unfolding of infinite Gaussian mixture model](https://mlsanity.com/paper/2202.06524/) (2022)
- [Reformulating Speaker Diarization as Community Detection With Emphasis On Topological Structure](https://mlsanity.com/paper/2204.12112/) (2022)
- [AG-LSEC: Audio Grounded Lexical Speaker Error Correction](https://mlsanity.com/paper/2406.17266/) (2024)
- [End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations](https://mlsanity.com/paper/2303.12002/) (2023)
- [Online speaker diarization of meetings guided by speech separation](https://mlsanity.com/paper/2402.00067/) (2024)
- [Character-aware audio-visual subtitling in context](https://mlsanity.com/paper/2410.11068/) (2024)
- [Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers](https://mlsanity.com/paper/2506.19875/) (2025)
- [Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network](https://mlsanity.com/paper/2309.08489/) (2023)
- [Speakers Unembedded: Embedding-free Approach to Long-form Neural Diarization](https://mlsanity.com/paper/2406.18679/) (2024)
- [SEAL: Speaker Error Correction using Acoustic-conditioned Large Language Models](https://mlsanity.com/paper/2501.08421/) (2025)
- [Probabilistic Fusion and Calibration of Neural Speaker Diarization Models](https://mlsanity.com/paper/2511.22696/) (2025)
- [Speaker Diarization using Two-pass Leave-One-Out Gaussian PLDA Clustering of DNN Embeddings](https://mlsanity.com/paper/2104.02469/) (2021)
- [Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework](https://mlsanity.com/paper/2406.16315/) (2024)
- [Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios](https://mlsanity.com/paper/2407.01317/) (2024)

## Abstract (arXiv)

Recent diarization technologies can be categorized into two approaches, i.e., clustering and end-to-end neural approaches, which have different pros and cons. The clustering-based approaches assign speaker labels to speech regions by clustering speaker embeddings such as x-vectors. While it can be seen as a current state-of-the-art approach that works for various challenging data with reasonable robustness and accuracy, it has a critical disadvantage that it cannot handle overlapped speech that is inevitable in natural conversational data. In contrast, the end-to-end neural diarization (EEND), which directly predicts diarization labels using a neural network, was devised to handle the overlapped speech. While the EEND, which can easily incorporate emerging deep-learning technologies, has started outperforming the x-vector clustering approach in some realistic database, it is difficult to make it work for `long' recordings (e.g., recordings longer than 10 minutes) because of, e.g., its huge memory consumption. Block-wise independent processing is also difficult because it poses an inter-block label permutation problem, i.e., an ambiguity of the speaker label assignments between blocks. In this paper, we propose a simple but effective hybrid diarization framework that works with overlapped speech and for long recordings containing an arbitrary number of speakers. It modifies the conventional EEND framework to simultaneously output global speaker embeddings so that speaker clustering can be performed across blocks to solve the permutation problem. With experiments based on simulated noisy reverberant 2-speaker meeting-like data, we show that the proposed framework works significantly better than the original EEND especially when the input data is long.