# Self-supervised speaker embeddings

arXiv: https://arxiv.org/abs/1904.03486 · HTML: https://mlsanity.com/paper/1904.03486/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2019.

## Built on by (most cited first, 12 of 12)

- [What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis](https://mlsanity.com/paper/2401.17632/) (2024)
- [Non-Contrastive Self-supervised Learning for Utterance-Level Information Extraction from Speech](https://mlsanity.com/paper/2208.05445/) (2022)
- [Non-Contrastive Self-Supervised Learning of Utterance-Level Speech Representations](https://mlsanity.com/paper/2208.05413/) (2022)
- [Experimenting with Additive Margins for Contrastive Self-Supervised Speaker Verification](https://mlsanity.com/paper/2306.03664/) (2023)
- [Additive Margin in Contrastive Self-Supervised Frameworks to Learn Discriminative Speaker Representations](https://mlsanity.com/paper/2404.14913/) (2024)
- [An Overview of Indian Spoken Language Recognition from Machine Learning Perspective](https://mlsanity.com/paper/2212.03812/) (2022)
- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)
- [Speaker embeddings by modeling channel-wise correlations](https://mlsanity.com/paper/2104.02571/) (2021)
- [The JHU submission to VoxSRC-21: Track 3](https://mlsanity.com/paper/2109.13425/) (2021)
- [Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification](https://mlsanity.com/paper/2112.08929/) (2021)
- [Towards the Next Frontier in Speech Representation Learning Using Disentanglement](https://mlsanity.com/paper/2407.02543/) (2024)
- [Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings](https://mlsanity.com/paper/2508.14115/) (2025)

## Abstract (arXiv)

Contrary to i-vectors, speaker embeddings such as x-vectors are incapable of leveraging unlabelled utterances, due to the classification loss over training speakers. In this paper, we explore an alternative training strategy to enable the use of unlabelled utterances in training. We propose to train speaker embedding extractors via reconstructing the frames of a target speech segment, given the inferred embedding of another speech segment of the same utterance. We do this by attaching to the standard speaker embedding extractor a decoder network, which we feed not merely with the speaker embedding, but also with the estimated phone sequence of the target frame sequence. The reconstruction loss can be used either as a single objective, or be combined with the standard speaker classification loss. In the latter case, it acts as a regularizer, encouraging generalizability to speakers unseen during training. In all cases, the proposed architectures are trained from scratch and in an end-to-end fashion. We demonstrate the benefits from the proposed approach on VoxCeleb and Speakers in the wild, and we report notable improvements over the baseline.