# DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding

arXiv: https://arxiv.org/abs/2105.13802 · HTML: https://mlsanity.com/paper/2105.13802/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds](https://mlsanity.com/paper/2010.13366/) (2020)
- [LSTM based Similarity Measurement with Spectral Clustering for Speaker Diarization](https://mlsanity.com/paper/1907.10393/) (2019)
- [Discriminative Neural Clustering for Speaker Diarisation](https://mlsanity.com/paper/1910.09703/) (2019)
- [Neural Speaker Diarization with Speaker-Wise Chain Rule](https://mlsanity.com/paper/2006.01796/) (2020)
- [Online End-to-End Neural Diarization with Speaker-Tracing Buffer](https://mlsanity.com/paper/2006.02616/) (2020)
- [End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification](https://mlsanity.com/paper/2003.02966/) (2020)
- [End-to-End Speaker Diarization as Post-Processing](https://mlsanity.com/paper/2012.10055/) (2020)

## Built on by (most cited first, 5 of 5)

- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Self-supervised Speaker Diarization](https://mlsanity.com/paper/2204.04166/) (2022)
- [End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations](https://mlsanity.com/paper/2303.12002/) (2023)
- [Speakers Unembedded: Embedding-free Approach to Long-form Neural Diarization](https://mlsanity.com/paper/2406.18679/) (2024)
- [Collar-aware Training for Streaming Speaker Change Detection in Broadcast Speech](https://mlsanity.com/paper/2205.07086/) (2022)

## Abstract (arXiv)

We introduce DIVE, an end-to-end speaker diarization algorithm. Our neural algorithm presents the diarization task as an iterative process: it repeatedly builds a representation for each speaker before predicting the voice activity of each speaker conditioned on the extracted representations. This strategy intrinsically resolves the speaker ordering ambiguity without requiring the classical permutation invariant training loss. In contrast with prior work, our model does not rely on pretrained speaker representations and optimizes all parameters of the system with a multi-speaker voice activity loss. Importantly, our loss explicitly excludes unreliable speaker turn boundaries from training, which is adapted to the standard collar-based Diarization Error Rate (DER) evaluation. Overall, these contributions yield a system redefining the state-of-the-art on the standard CALLHOME benchmark, with 6.7% DER compared to 7.8% for the best alternative.