# Residual Information in Deep Speaker Embedding Architectures

arXiv: https://arxiv.org/abs/2302.02742 · HTML: https://mlsanity.com/paper/2302.02742/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 24 of 24)

- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context](https://mlsanity.com/paper/2110.04410/) (2021)
- [Speaker Recognition Based on Deep Learning: An Overview](https://mlsanity.com/paper/2012.00931/) (2020)
- [SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model](https://mlsanity.com/paper/2104.05557/) (2021)
- [ECAPA-TDNN Embeddings for Speaker Diarization](https://mlsanity.com/paper/2104.01466/) (2021)
- [Fine-tuning wav2vec2 for speaker recognition](https://mlsanity.com/paper/2109.15053/) (2021)
- [RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification](https://mlsanity.com/paper/1904.08104/) (2019)
- [The ins and outs of speaker recognition: lessons from VoxSRC 2020](https://mlsanity.com/paper/2010.15809/) (2020)
- [Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Data](https://mlsanity.com/paper/2009.14399/) (2020)
- [The IDLAB VoxCeleb Speaker Recognition Challenge 2020 System Description](https://mlsanity.com/paper/2010.12468/) (2020)
- [Speaker Generation](https://mlsanity.com/paper/2111.05095/) (2021)
- [SpeakerNet: 1D Depth-wise Separable Convolutional Network for Text-Independent Speaker Recognition and Verification](https://mlsanity.com/paper/2010.12653/) (2020)
- [Whispered and Lombard Neural Speech Synthesis](https://mlsanity.com/paper/2101.05313/) (2021)
- [Multi-Task Learning with High-Order Statistics for X-vector based Text-Independent Speaker Verification](https://mlsanity.com/paper/1903.12058/) (2019)
- [The Kriston AI System for the VoxCeleb Speaker Recognition Challenge 2022](https://mlsanity.com/paper/2209.11433/) (2022)
- [The ReturnZero System for VoxCeleb Speaker Recognition Challenge 2022](https://mlsanity.com/paper/2209.10147/) (2022)

## Built on by (most cited first, 2 of 2)

- [An analysis on the effects of speaker embedding choice in non auto-regressive TTS](https://mlsanity.com/paper/2307.09898/) (2023)
- [Residual Speech Embeddings for Tone Classification: Removing Linguistic Content to Enhance Paralinguistic Analysis](https://mlsanity.com/paper/2502.19387/) (2025)

## Abstract (arXiv)

Speaker embeddings represent a means to extract representative vectorial representations from a speech signal such that the representation pertains to the speaker identity alone. The embeddings are commonly used to classify and discriminate between different speakers. However, there is no objective measure to evaluate the ability of a speaker embedding to disentangle the speaker identity from the other speech characteristics. This means that the embeddings are far from ideal, highly dependent on the training corpus and still include a degree of residual information pertaining to factors such as linguistic content, recording conditions or speaking style of the utterance. This paper introduces an analysis over six sets of speaker embeddings extracted with some of the most recent and high-performing DNN architectures, and in particular, the degree to which they are able to truly disentangle the speaker identity from the speech signal. To correctly evaluate the architectures, a large multi-speaker parallel speech dataset is used. The dataset includes 46 speakers uttering the same set of prompts, recorded in either a professional studio or their home environments. The analysis looks into the intra- and inter-speaker similarity measures computed over the different embedding sets, as well as if simple classification and regression methods are able to extract several residual information factors from the speaker embeddings. The results show that the discriminative power of the analyzed embeddings is very high, yet across all the analyzed architectures, residual information is still present in the representations in the form of a high correlation to the recording conditions, linguistic contents and utterance duration.