# Speaker Diarization with LSTM

arXiv: https://arxiv.org/abs/1710.10468 · HTML: https://mlsanity.com/paper/1710.10468/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2017.

## Built on by (most cited first, 30 of 73)

- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [A Light Weight Model for Active Speaker Detection](https://mlsanity.com/paper/2303.04439/) (2023)
- [MAAS: Multi-modal Assignation for Active Speaker Detection](https://mlsanity.com/paper/2101.03682/) (2021)
- [Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection](https://mlsanity.com/paper/2109.11641/) (2021)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [LoCoNet: Long-Short Context Network for Active Speaker Detection](https://mlsanity.com/paper/2301.08237/) (2023)
- [Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization](https://mlsanity.com/paper/2106.14118/) (2021)
- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correction](https://mlsanity.com/paper/2306.09313/) (2023)
- [A Real-time Speaker Diarization System Based on Spatial Spectrum](https://mlsanity.com/paper/2107.09321/) (2021)
- [Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering](https://mlsanity.com/paper/2210.13690/) (2022)
- [VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2302.10248/) (2023)
- [The VoxCeleb Speaker Recognition Challenge: A Retrospective](https://mlsanity.com/paper/2408.14886/) (2024)
- [Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT](https://mlsanity.com/paper/2205.07180/) (2022)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling](https://mlsanity.com/paper/2401.12039/) (2024)
- [End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings](https://mlsanity.com/paper/2105.02096/) (2021)
- [Using Active Speaker Faces for Diarization in TV shows](https://mlsanity.com/paper/2203.15961/) (2022)
- [Supervised Hierarchical Clustering using Graph Neural Networks for Speaker Diarization](https://mlsanity.com/paper/2302.12716/) (2023)
- [A Benchmark for Multi-speaker Anonymization](https://mlsanity.com/paper/2407.05608/) (2024)
- [Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustness](https://mlsanity.com/paper/2505.21954/) (2025)
- [DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding](https://mlsanity.com/paper/2105.13802/) (2021)
- [Directed Speech Separation for Automatic Speech Recognition of Long Form Conversational Speech](https://mlsanity.com/paper/2112.05863/) (2021)
- [PrecTime: A Deep Learning Architecture for Precise Time Series Segmentation in Industrial Manufacturing Operations](https://mlsanity.com/paper/2302.10182/) (2023)
- [Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization](https://mlsanity.com/paper/2305.12927/) (2023)
- [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024)
- [MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models](https://mlsanity.com/paper/2411.18152/) (2024)
- [Reformulating Speaker Diarization as Community Detection With Emphasis On Topological Structure](https://mlsanity.com/paper/2204.12112/) (2022)
- [Neural Diarization with Non-autoregressive Intermediate Attractors](https://mlsanity.com/paper/2303.06806/) (2023)

## Abstract (arXiv)

For many years, i-vector based audio embedding techniques were the dominant approach for speaker verification and speaker diarization applications. However, mirroring the rise of deep learning in various domains, neural network based audio embeddings, also known as d-vectors, have consistently demonstrated superior speaker verification performance. In this paper, we build on the success of d-vector based speaker verification systems to develop a new d-vector based approach to speaker diarization. Specifically, we combine LSTM-based d-vector audio embeddings with recent work in non-parametric clustering to obtain a state-of-the-art speaker diarization system. Our system is evaluated on three standard public datasets, suggesting that d-vector based diarization systems offer significant advantages over traditional i-vector based systems. We achieved a 12.0% diarization error rate on NIST SRE 2000 CALLHOME, while our model is trained with out-of-domain data from voice search logs.