# High-resolution embedding extractor for speaker diarisation

arXiv: https://arxiv.org/abs/2211.04060 · HTML: https://mlsanity.com/paper/2211.04060/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [The Third DIHARD Diarization Challenge](https://mlsanity.com/paper/2012.01477/) (2020)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [Recognizing Multi-talker Speech with Permutation Invariant Training](https://mlsanity.com/paper/1704.01985/) (2017)
- [Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection](https://mlsanity.com/paper/2109.11641/) (2021)
- [VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2012.06867/) (2020)
- [Multi-scale Speaker Diarization with Dynamic Scale Weighting](https://mlsanity.com/paper/2203.15974/) (2022)
- [Multi-Scale Speaker Diarization With Neural Affinity Score Fusion](https://mlsanity.com/paper/2011.10527/) (2020)
- [VoxSRC 2021: The Third VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2201.04583/) (2022)
- [USTC-NELSLIP System Description for DIHARD-III Challenge](https://mlsanity.com/paper/2103.10661/) (2021)
- [Multi-scale speaker embedding-based graph attention networks for speaker diarisation](https://mlsanity.com/paper/2110.03361/) (2021)
- [BUT System Description for DIHARD Speech Diarization Challenge 2019](https://mlsanity.com/paper/1910.08847/) (2019)
- [Adapting Speaker Embeddings for Speaker Diarisation](https://mlsanity.com/paper/2104.02879/) (2021)
- [The Kriston AI System for the VoxCeleb Speaker Recognition Challenge 2022](https://mlsanity.com/paper/2209.11433/) (2022)

## Built on by (most cited first, 1 of 1)

- [Speaker and Language Change Detection using Wav2vec2 and Whisper](https://mlsanity.com/paper/2302.09381/) (2023)

## Abstract (arXiv)

Speaker embedding extractors significantly influence the performance of clustering-based speaker diarisation systems. Conventionally, only one embedding is extracted from each speech segment. However, because of the sliding window approach, a segment easily includes two or more speakers owing to speaker change points. This study proposes a novel embedding extractor architecture, referred to as a high-resolution embedding extractor (HEE), which extracts multiple high-resolution embeddings from each speech segment. Hee consists of a feature-map extractor and an enhancer, where the enhancer with the self-attention mechanism is the key to success. The enhancer of HEE replaces the aggregation process; instead of a global pooling layer, the enhancer combines relative information to each frame via attention leveraging the global context. Extracted dense frame-level embeddings can each represent a speaker. Thus, multiple speakers can be represented by different frame-level features in each segment. We also propose an artificially generating mixture data training framework to train the proposed HEE. Through experiments on five evaluation sets, including four public datasets, the proposed HEE demonstrates at least 10% improvement on each evaluation set, except for one dataset, which we analyse that rapid speaker changes less exist.