# From Modular to End-to-End Speaker Diarization

arXiv: https://arxiv.org/abs/2407.08752 · HTML: https://mlsanity.com/paper/2407.08752/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 290 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [BUT System Description to VoxCeleb Speaker Recognition Challenge 2019](https://mlsanity.com/paper/1910.12592/) (2019)
- [VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2012.06867/) (2020)
- [Discriminative Neural Clustering for Speaker Diarisation](https://mlsanity.com/paper/1910.09703/) (2019)
- [Neural Speaker Diarization with Speaker-Wise Chain Rule](https://mlsanity.com/paper/2006.01796/) (2020)
- [The Hitachi-JHU DIHARD III System: Competitive End-to-End Neural Diarization and X-Vector Clustering Systems Combined by DOVER-Lap](https://mlsanity.com/paper/2102.01363/) (2021)
- [USTC-NELSLIP System Description for DIHARD-III Challenge](https://mlsanity.com/paper/2103.10661/) (2021)
- [Attention-based Encoder-Decoder End-to-End Neural Diarization with Embedding Enhancer](https://mlsanity.com/paper/2309.06672/) (2023)
- [DiaPer: End-to-End Neural Diarization with Perceiver-Based Attractors](https://mlsanity.com/paper/2312.04324/) (2023)
- [Profile-Error-Tolerant Target-Speaker Voice Activity Detection](https://mlsanity.com/paper/2309.12521/) (2023)
- [BER: Balanced Error Rate For Speaker Diarization](https://mlsanity.com/paper/2211.04304/) (2022)
- [Semi-supervised multi-channel speaker diarization with cross-channel attention](https://mlsanity.com/paper/2307.08688/) (2023)
- [Speaker Embedding-aware Neural Diarization: an Efficient Framework for Overlapping Speech Diarization in Meeting Scenarios](https://mlsanity.com/paper/2203.09767/) (2022)
- [Discriminative Training of VBx Diarization](https://mlsanity.com/paper/2310.02732/) (2023)
- [Speaker Embedding-aware Neural Diarization for Flexible Number of Speakers with Textual Information](https://mlsanity.com/paper/2111.13694/) (2021)
- [Late Audio-Visual Fusion for In-The-Wild Speaker Diarization](https://mlsanity.com/paper/2211.01299/) (2022)

## Abstract (arXiv)

Speaker diarization is usually referred to as the task that determines ``who spoke when'' in a recording. Until a few years ago, all competitive approaches were modular. Systems based on this framework reached state-of-the-art performance in most scenarios but had major difficulties dealing with overlapped speech. More recently, the advent of end-to-end models, capable of dealing with all aspects of speaker diarization with a single model and better performing regarding overlapped speech, has brought high levels of attention. This thesis is framed during a period of co-existence of these two trends. We describe a system based on a Bayesian hidden Markov model used to cluster x-vectors (speaker embeddings obtained with a neural network), known as VBx, which has shown remarkable performance on different datasets and challenges. We comment on its advantages and limitations and evaluate results on different relevant corpora. Then, we move towards end-to-end neural diarization (EEND) methods. Due to the need for large training sets for training these models and the lack of manually annotated diarization data in sufficient quantities, the compromise solution consists in generating training data artificially. We describe an approach for generating synthetic data which resembles real conversations in terms of speaker turns and overlaps. We show how this method generating ``simulated conversations'' allows for better performance than using a previously proposed method for creating ``simulated mixtures'' when training the popular EEND with encoder-decoder attractors (EEND-EDA). We also propose a new EEND-based model, which we call DiaPer, and show that it can perform better than EEND-EDA, especially when dealing with many speakers and handling overlapped speech. Finally, we compare both VBx-based and DiaPer systems on a wide variety of corpora and comment on the advantages of each technique.