# Disentangling Voice and Content with Self-Supervision for Speaker Recognition

arXiv: https://arxiv.org/abs/2310.01128 · HTML: https://mlsanity.com/paper/2310.01128/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 104 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data](https://mlsanity.com/paper/1709.07902/) (2017)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces](https://mlsanity.com/paper/1905.07357/) (2019)
- [Speaker Recognition Based on Deep Learning: An Overview](https://mlsanity.com/paper/2012.00931/) (2020)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [End-to-End Text-Dependent Speaker Verification](https://mlsanity.com/paper/1509.08062/) (2015)

## Built on by (most cited first, 13 of 13)

- [Nes2Net: A Lightweight Nested Architecture for Foundation Model Driven Speech Anti-spoofing](https://mlsanity.com/paper/2504.05657/) (2025)
- [Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detection](https://mlsanity.com/paper/2406.10283/) (2024)
- [How Do Neural Spoofing Countermeasures Detect Partially Spoofed Audio?](https://mlsanity.com/paper/2406.02483/) (2024)
- [Speech Foundation Model Ensembles for the Controlled Singing Voice Deepfake Detection (CtrSVDD) Challenge 2024](https://mlsanity.com/paper/2409.02302/) (2024)
- [Towards Quantifying and Reducing Language Mismatch Effects in Cross-Lingual Speech Anti-Spoofing](https://mlsanity.com/paper/2409.08346/) (2024)
- [Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision](https://mlsanity.com/paper/2406.11169/) (2024)
- [Scores Know Bobs Voice: Speaker Impersonation Attack](https://mlsanity.com/paper/2603.02781/) (2026)
- [Pushing the Frontiers of Self-Distillation Prototypes Network with Dimension Regularization and Score Normalization](https://mlsanity.com/paper/2505.13826/) (2025)
- [Interpolating Speaker Identities in Embedding Space for Data Expansion](https://mlsanity.com/paper/2508.19210/) (2025)
- [Cosine Scoring with Uncertainty for Neural Speaker Embedding](https://mlsanity.com/paper/2403.06404/) (2024)
- [Robust Generative Audio Quality Assessment: Disentangling Quality from Spurious Correlations](https://mlsanity.com/paper/2603.16201/) (2026)
- [Listening Alone, Understanding Together: Collaborative Context Recovery for Privacy-Aware AI](https://mlsanity.com/paper/2604.13348/) (2026)
- [DEFT: Disentanglement-Enhanced Fine-Tuning for EEG Foundation Models](https://mlsanity.com/paper/2610.14497/) (2026)

## Abstract (arXiv)

For speaker recognition, it is difficult to extract an accurate speaker representation from speech because of its mixture of speaker traits and content. This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is realized with the use of three Gaussian inference layers, each consisting of a learnable transition model that extracts distinct speech components. Notably, a strengthened transition model is specifically designed to model complex speech dynamics. We also propose a self-supervision method to dynamically disentangle content without the use of labels other than speaker identities. The efficacy of the proposed framework is validated via experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF, respectively. Since neither additional model training nor data is specifically needed, it is easily applicable in practical use.