# End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors

arXiv: https://arxiv.org/abs/2005.09921 · HTML: https://mlsanity.com/paper/2005.09921/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 30 of 71)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2110.03151/) (2021)
- [VoxCeleb Enrichment for Age and Gender Recognition](https://mlsanity.com/paper/2109.13510/) (2021)
- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems](https://mlsanity.com/paper/2409.06656/) (2024)
- [BERTraffic: BERT-based Joint Speaker Role and Speaker Change Detection for Air Traffic Control Communications](https://mlsanity.com/paper/2110.05781/) (2021)
- [Multi-scale Speaker Diarization with Dynamic Scale Weighting](https://mlsanity.com/paper/2203.15974/) (2022)
- [Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detection](https://mlsanity.com/paper/2303.08019/) (2023)
- [The VoxCeleb Speaker Recognition Challenge: A Retrospective](https://mlsanity.com/paper/2408.14886/) (2024)
- [The Hitachi-JHU DIHARD III System: Competitive End-to-End Neural Diarization and X-Vector Clustering Systems Combined by DOVER-Lap](https://mlsanity.com/paper/2102.01363/) (2021)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling](https://mlsanity.com/paper/2401.12039/) (2024)
- [End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings](https://mlsanity.com/paper/2105.02096/) (2021)
- [Multi-scale speaker embedding-based graph attention networks for speaker diarisation](https://mlsanity.com/paper/2110.03361/) (2021)
- [Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.11232/) (2022)
- [Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization](https://mlsanity.com/paper/2208.13085/) (2022)
- [Supervised Hierarchical Clustering using Graph Neural Networks for Speaker Diarization](https://mlsanity.com/paper/2302.12716/) (2023)
- [Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator](https://mlsanity.com/paper/2305.16263/) (2023)
- [From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understanding](https://mlsanity.com/paper/2507.02790/) (2025)
- [DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding](https://mlsanity.com/paper/2105.13802/) (2021)
- [A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2203.16834/) (2022)
- [In search of strong embedding extractors for speaker diarisation](https://mlsanity.com/paper/2210.14682/) (2022)
- [Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization](https://mlsanity.com/paper/2305.12927/) (2023)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [ASR-Aware End-to-end Neural Diarization](https://mlsanity.com/paper/2202.01286/) (2022)
- [Focus Agent: LLM-Powered Virtual Focus Group](https://mlsanity.com/paper/2409.01907/) (2024)

## Abstract (arXiv)

End-to-end speaker diarization for an unknown number of speakers is addressed in this paper. Recently proposed end-to-end speaker diarization outperformed conventional clustering-based speaker diarization, but it has one drawback: it is less flexible in terms of the number of speakers. This paper proposes a method for encoder-decoder based attractor calculation (EDA), which first generates a flexible number of attractors from a speech embedding sequence. Then, the generated multiple attractors are multiplied by the speech embedding sequence to produce the same number of speaker activities. The speech embedding sequence is extracted using the conventional self-attentive end-to-end neural speaker diarization (SA-EEND) network. In a two-speaker condition, our method achieved a 2.69 % diarization error rate (DER) on simulated mixtures and a 8.07 % DER on the two-speaker subset of CALLHOME, while vanilla SA-EEND attained 4.56 % and 9.54 %, respectively. In unknown numbers of speakers conditions, our method attained a 15.29 % DER on CALLHOME, while the x-vector-based clustering method achieved a 19.43 % DER.