# In search of strong embedding extractors for speaker diarisation

arXiv: https://arxiv.org/abs/2210.14682 · HTML: https://mlsanity.com/paper/2210.14682/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks](https://mlsanity.com/paper/2012.14952/) (2020)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [Pushing the limits of raw waveform speaker recognition](https://mlsanity.com/paper/2203.08488/) (2022)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection](https://mlsanity.com/paper/1910.11646/) (2019)
- [From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.00890/) (2022)
- [Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy](https://mlsanity.com/paper/1905.00268/) (2019)
- [Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2010.11458/) (2020)
- [Delving into VoxCeleb: environment invariant speaker recognition](https://mlsanity.com/paper/1910.11238/) (2019)
- [Multi-scale Speaker Diarization with Dynamic Scale Weighting](https://mlsanity.com/paper/2203.15974/) (2022)
- [Multi-class Spectral Clustering with Overlaps for Speaker Diarization](https://mlsanity.com/paper/2011.02900/) (2020)
- [End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings](https://mlsanity.com/paper/2105.02096/) (2021)
- [Multi-scale speaker embedding-based graph attention networks for speaker diarisation](https://mlsanity.com/paper/2110.03361/) (2021)
- [Adapting Speaker Embeddings for Speaker Diarisation](https://mlsanity.com/paper/2104.02879/) (2021)
- [Large-scale learning of generalised representations for speaker recognition](https://mlsanity.com/paper/2210.10985/) (2022)

## Built on by (most cited first, 5 of 5)

- [ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models](https://mlsanity.com/paper/2401.17230/) (2024)
- [VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2302.10248/) (2023)
- [The VoxCeleb Speaker Recognition Challenge: A Retrospective](https://mlsanity.com/paper/2408.14886/) (2024)
- [Encoder-decoder multimodal speaker change detection](https://mlsanity.com/paper/2306.00680/) (2023)
- [Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings](https://mlsanity.com/paper/2508.14115/) (2025)

## Abstract (arXiv)

Speaker embedding extractors (EEs), which map input audio to a speaker discriminant latent space, are of paramount importance in speaker diarisation. However, there are several challenges when adopting EEs for diarisation, from which we tackle two key problems. First, the evaluation is not straightforward because the features required for better performance differ between speaker verification and diarisation. We show that better performance on widely adopted speaker verification evaluation protocols does not lead to better diarisation performance. Second, embedding extractors have not seen utterances in which multiple speakers exist. These inputs are inevitably present in speaker diarisation because of overlapped speech and speaker changes; they degrade the performance. To mitigate the first problem, we generate speaker verification evaluation protocols that mimic the diarisation scenario better. We propose two data augmentation techniques to alleviate the second problem, making embedding extractors aware of overlapped speech or speaker change input. One technique generates overlapped speech segments, and the other generates segments where two speakers utter sequentially. Extensive experimental results using three state-of-the-art speaker embedding extractors demonstrate that both proposed approaches are effective.