# Multi-scale Speaker Diarization with Dynamic Scale Weighting

arXiv: https://arxiv.org/abs/2203.15974 · HTML: https://mlsanity.com/paper/2203.15974/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks](https://mlsanity.com/paper/2012.14952/) (2020)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context](https://mlsanity.com/paper/2110.04410/) (2021)
- [ECAPA-TDNN Embeddings for Speaker Diarization](https://mlsanity.com/paper/2104.01466/) (2021)
- [Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengap](https://mlsanity.com/paper/2003.02405/) (2020)
- [Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds](https://mlsanity.com/paper/2010.13366/) (2020)
- [Multi-Scale Speaker Diarization With Neural Affinity Score Fusion](https://mlsanity.com/paper/2011.10527/) (2020)
- [Third DIHARD Challenge Evaluation Plan](https://mlsanity.com/paper/2006.05815/) (2020)

## Built on by (most cited first, 11 of 11)

- [NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription](https://mlsanity.com/paper/2401.08887/) (2024)
- [Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems](https://mlsanity.com/paper/2409.06656/) (2024)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [LLM-based speaker diarization correction: A generalizable approach](https://mlsanity.com/paper/2406.04927/) (2024)
- [In search of strong embedding extractors for speaker diarisation](https://mlsanity.com/paper/2210.14682/) (2022)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [High-resolution embedding extractor for speaker diarisation](https://mlsanity.com/paper/2211.04060/) (2022)
- [Investigating the Effects of Large-Scale Pseudo-Stereo Data and Different Speech Foundation Model on Dialogue Generative Spoken Language Model](https://mlsanity.com/paper/2407.01911/) (2024)
- [An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutions](https://mlsanity.com/paper/2410.23955/) (2024)
- [Model-Agnostic and Language-Agnostic Voice Pipeline Improvement for the Agriculture Domain](https://mlsanity.com/paper/2609.20504/) (2026)
- [Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization](https://mlsanity.com/paper/2609.39162/) (2026)

## Abstract (arXiv)

Speaker diarization systems are challenged by a trade-off between the temporal resolution and the fidelity of the speaker representation. By obtaining a superior temporal resolution with an enhanced accuracy, a multi-scale approach is a way to cope with such a trade-off. In this paper, we propose a more advanced multi-scale diarization system based on a multi-scale diarization decoder. There are two main contributions in this study that significantly improve the diarization performance. First, we use multi-scale clustering as an initialization to estimate the number of speakers and obtain the average speaker representation vector for each speaker and each scale. Next, we propose the use of 1-D convolutional neural networks that dynamically determine the importance of each scale at each time step. To handle a variable number of speakers and overlapping speech, the proposed system can estimate the number of existing speakers. Our proposed system achieves a state-of-art performance on the CALLHOME and AMI MixHeadset datasets, with 3.92% and 1.05% diarization error rates, respectively.