# FASTDIAR: Frame-level speaker encoder for Streaming Diarization

arXiv: https://arxiv.org/abs/2610.02941 · HTML: https://mlsanity.com/paper/2610.02941/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 29 of 29)

- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Dataset](https://mlsanity.com/paper/2203.16844/) (2022)
- [Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection](https://mlsanity.com/paper/2109.11641/) (2021)
- [Reshape Dimensions Network for Speaker Recognition](https://mlsanity.com/paper/2407.18223/) (2024)
- [VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark](https://mlsanity.com/paper/2407.11510/) (2024)
- [Leveraging Self-Supervised Learning for Speaker Diarization](https://mlsanity.com/paper/2409.09408/) (2024)
- [NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription](https://mlsanity.com/paper/2401.08887/) (2024)
- [Overlap-aware low-latency online speaker diarization based on end-to-end local segmentation](https://mlsanity.com/paper/2109.06483/) (2021)
- [Online End-to-End Neural Diarization with Speaker-Tracing Buffer](https://mlsanity.com/paper/2006.02616/) (2020)
- [BW-EDA-EEND: Streaming End-to-End Neural Speaker Diarization for a Variable Number of Speakers](https://mlsanity.com/paper/2011.02678/) (2020)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering](https://mlsanity.com/paper/2507.18446/) (2025)
- [Short utterance compensation in speaker verification via cosine-based teacher-student learning of speaker embeddings](https://mlsanity.com/paper/1810.10884/) (2018)
- [Links: A High-Dimensional Online Clustering Method](https://mlsanity.com/paper/1801.10123/) (2018)
- [Frame-wise and overlap-robust speaker embeddings for meeting diarization](https://mlsanity.com/paper/2306.00625/) (2023)
- [LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization](https://mlsanity.com/paper/2409.00819/) (2024)
- [Frame-wise streaming end-to-end speaker diarization with non-autoregressive self-attention-based attractors](https://mlsanity.com/paper/2309.13916/) (2023)
- [ReDimNet2: Scaling Speaker Verification via Time-Pooled Dimension Reshaping](https://mlsanity.com/paper/2603.11841/) (2026)
- [LS-EEND: Long-Form Streaming End-to-End Neural Diarization with Online Attractor Extraction](https://mlsanity.com/paper/2410.06670/) (2024)
- [Investigating Confidence Estimation Measures for Speaker Diarization](https://mlsanity.com/paper/2406.17124/) (2024)

## Abstract (arXiv)

Real-time conversational agents require speaker diarization that streams and runs on a CPU. Most systems apply an utterance-level speaker encoder to short, heavily overlapping chunks, which wastes computation and leaves the model optimized for the wrong task. We instead turn a state-of-the-art speaker recognition architecture into a causal frame-level encoder that reads the stream once and emits one embedding every 80~ms from a bounded two-second window of past audio, and pair it with online clustering that gates every update on the self-similarity of the stream. Trained only by distillation from an utterance-level teacher on simulated and out-of-domain mixtures, and evaluated with one fixed set of hyperparameters, the system is the most accurate streaming diarizer on low-overlap benchmarks at sub-second latency, degrades far less than cache-based systems as the number of speakers grows, and runs five times faster than real time on a single CPU thread.