# Exploring Disentanglement with Multilingual and Monolingual VQ-VAE

arXiv: https://arxiv.org/abs/2105.01573 · HTML: https://mlsanity.com/paper/2105.01573/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2021.

## Built on by (most cited first, 4 of 4)

- [EEG2Vec: Learning Affective EEG Representations via Variational Autoencoders](https://mlsanity.com/paper/2207.08002/) (2022)
- [Mix and Match: An Empirical Study on Training Corpus Composition for Polyglot Text-To-Speech (TTS)](https://mlsanity.com/paper/2207.01507/) (2022)
- [Exploratory Evaluation of Speech Content Masking](https://mlsanity.com/paper/2401.03936/) (2024)
- [Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling](https://mlsanity.com/paper/2409.08664/) (2024)

## Abstract (arXiv)

This work examines the content and usefulness of disentangled phone and speaker representations from two separately trained VQ-VAE systems: one trained on multilingual data and another trained on monolingual data. We explore the multi- and monolingual models using four small proof-of-concept tasks: copy-synthesis, voice transformation, linguistic code-switching, and content-based privacy masking. From these tasks, we reflect on how disentangled phone and speaker representations can be used to manipulate speech in a meaningful way. Our experiments demonstrate that the VQ representations are suitable for these tasks, including creating new voices by mixing speaker representations together. We also present our novel technique to conceal the content of targeted words within an utterance by manipulating phone VQ codes, while retaining speaker identity and intelligibility of surrounding words. Finally, we discuss recommendations for further increasing the viability of disentangled representations.