# Multi-scale speaker embedding-based graph attention networks for speaker diarisation

arXiv: https://arxiv.org/abs/2110.03361 · HTML: https://mlsanity.com/paper/2110.03361/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [End-to-End Spectro-Temporal Graph Attention Networks for Speaker Verification Anti-Spoofing and Speech Deepfake Detection](https://mlsanity.com/paper/2107.12710/) (2021)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [The Second DIHARD Diarization Challenge: Dataset, task, and baselines](https://mlsanity.com/paper/1906.07839/) (2019)
- [The ins and outs of speaker recognition: lessons from VoxSRC 2020](https://mlsanity.com/paper/2010.15809/) (2020)
- [Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2010.11458/) (2020)
- [Graph Attention Networks for Anti-Spoofing](https://mlsanity.com/paper/2104.03654/) (2021)
- [Multi-Scale Speaker Diarization With Neural Affinity Score Fusion](https://mlsanity.com/paper/2011.10527/) (2020)
- [Three-class Overlapped Speech Detection using a Convolutional Recurrent Neural Network](https://mlsanity.com/paper/2104.02878/) (2021)
- [USTC-NELSLIP System Description for DIHARD-III Challenge](https://mlsanity.com/paper/2103.10661/) (2021)
- [The ByteDance Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2021](https://mlsanity.com/paper/2109.02047/) (2021)
- [Graph Attention Networks for Speaker Verification](https://mlsanity.com/paper/2010.11543/) (2020)
- [BUT System Description for DIHARD Speech Diarization Challenge 2019](https://mlsanity.com/paper/1910.08847/) (2019)
- [Adapting Speaker Embeddings for Speaker Diarisation](https://mlsanity.com/paper/2104.02879/) (2021)

## Built on by (most cited first, 7 of 7)

- [In search of strong embedding extractors for speaker diarisation](https://mlsanity.com/paper/2210.14682/) (2022)
- [ATGNN: Audio Tagging Graph Neural Network](https://mlsanity.com/paper/2311.01526/) (2023)
- [High-resolution embedding extractor for speaker diarisation](https://mlsanity.com/paper/2211.04060/) (2022)
- [Encoder-decoder multimodal speaker change detection](https://mlsanity.com/paper/2306.00680/) (2023)
- [A Universal Knowledge Model and Cognitive Architecture for Prototyping AGI](https://mlsanity.com/paper/2401.06256/) (2024)
- [Geodesic interpolation of frame-wise speaker embeddings for the diarization of meeting scenarios](https://mlsanity.com/paper/2401.03963/) (2024)
- [Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios](https://mlsanity.com/paper/2407.01317/) (2024)

## Abstract (arXiv)

The objective of this work is effective speaker diarisation using multi-scale speaker embeddings. Typically, there is a trade-off between the ability to recognise short speaker segments and the discriminative power of the embedding, according to the segment length used for embedding extraction. To this end, recent works have proposed the use of multi-scale embeddings where segments with varying lengths are used. However, the scores are combined using a weighted summation scheme where the weights are fixed after the training phase, whereas the importance of segment lengths can differ with in a single session. To address this issue, we present three key contributions in this paper: (1) we propose graph attention networks for multi-scale speaker diarisation; (2) we design scale indicators to utilise scale information of each embedding; (3) we adapt the attention-based aggregation to utilise a pre-computed affinity matrix from multi-scale embeddings. We demonstrate the effectiveness of our method in various datasets where the speaker confusion which constitutes the primary metric drops over 10% in average relative compared to the baseline.