# Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm

arXiv: https://arxiv.org/abs/2010.10727 · HTML: https://mlsanity.com/paper/2010.10727/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [Disentangled Feature Learning for Real-Time Neural Speech Coding](https://mlsanity.com/paper/2211.11960/) (2022)
- [End-to-End Neural Speech Coding for Real-Time Communications](https://mlsanity.com/paper/2201.09429/) (2022)
- [Latent-Domain Predictive Neural Speech Coding](https://mlsanity.com/paper/2207.08363/) (2022)
- [MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding](https://mlsanity.com/paper/2507.06071/) (2025)
- [Dictionary Attacks on Speaker Verification](https://mlsanity.com/paper/2204.11304/) (2022)
- [StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis](https://mlsanity.com/paper/2312.12181/) (2023)
- [Exploratory Evaluation of Speech Content Masking](https://mlsanity.com/paper/2401.03936/) (2024)
- [OCTOPUS: Overcoming Performance andPrivatization Bottlenecks in Distributed Learning](https://mlsanity.com/paper/2105.00602/) (2021)
- [Disentanglement of Latent Representations via Causal Interventions](https://mlsanity.com/paper/2302.00869/) (2023)
- [Improving Voice Quality in Speech Anonymization With Just Perception-Informed Losses](https://mlsanity.com/paper/2410.15499/) (2024)
- [Vector Quantized-Elites: Unsupervised and Problem-Agnostic Quality-Diversity Optimization](https://mlsanity.com/paper/2504.08057/) (2025)
- [Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance](https://mlsanity.com/paper/2106.13479/) (2021)

## Abstract (arXiv)

We present a new approach to disentangle speaker voice and phone content by introducing new components to the VQ-VAE architecture for speech synthesis. The original VQ-VAE does not generalize well to unseen speakers or content. To alleviate this problem, we have incorporated a speaker encoder and speaker VQ codebook that learns global speaker characteristics entirely separate from the existing sub-phone codebooks. We also compare two training methods: self-supervised with global conditions and semi-supervised with speaker labels. Adding a speaker VQ component improves objective measures of speech synthesis quality (estimated MOS, speaker similarity, ASR-based intelligibility) and provides learned representations that are meaningful. Our speaker VQ codebook indices can be used in a simple speaker diarization task and perform slightly better than an x-vector baseline. Additionally, phones can be recognized from sub-phone VQ codebook indices in our semi-supervised VQ-VAE better than self-supervised with global conditions.