# Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning

arXiv: https://arxiv.org/abs/2103.09420 · HTML: https://mlsanity.com/paper/2103.09420/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2021.

## Built on by (most cited first, 30 of 32)

- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models](https://mlsanity.com/paper/2206.02246/) (2022)
- [Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units](https://mlsanity.com/paper/2212.09730/) (2022)
- [StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching](https://mlsanity.com/paper/2412.04724/) (2024)
- [Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain](https://mlsanity.com/paper/2305.03256/) (2023)
- [Learning the Beauty in Songs: Neural Singing Voice Beautifier](https://mlsanity.com/paper/2202.13277/) (2022)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Voice Attribute Editing with Text Prompt](https://mlsanity.com/paper/2404.08857/) (2024)
- [Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignment](https://mlsanity.com/paper/2309.09470/) (2023)
- [Many-to-Many Voice Conversion based Feature Disentanglement using Variational Autoencoder](https://mlsanity.com/paper/2107.06642/) (2021)
- [Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features](https://mlsanity.com/paper/2112.04424/) (2021)
- [Semantic Feature Extraction for Generalized Zero-shot Learning](https://mlsanity.com/paper/2112.14478/) (2021)
- [So Different Yet So Alike! Constrained Unsupervised Text Style Transfer](https://mlsanity.com/paper/2205.04093/) (2022)
- [Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE](https://mlsanity.com/paper/2206.02512/) (2022)
- [Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance](https://mlsanity.com/paper/2512.23461/) (2025)
- [Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph](https://mlsanity.com/paper/2202.12307/) (2022)
- [SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machines](https://mlsanity.com/paper/2111.03811/) (2021)
- [Learning Disentangled Representation with Mutual Information Maximization for Real-Time UAV Tracking](https://mlsanity.com/paper/2308.10262/) (2023)
- [Tubes Among Us: Analog Attack on Automatic Speaker Identification](https://mlsanity.com/paper/2202.02751/) (2022)
- [Replacing Language Model for Style Transfer](https://mlsanity.com/paper/2211.07343/) (2022)
- [SLMGAN: Exploiting Speech Language Model Representations for Unsupervised Zero-Shot Voice Conversion in GANs](https://mlsanity.com/paper/2307.09435/) (2023)
- [Read the Room: Adapting a Robot's Voice to Ambient and Social Contexts](https://mlsanity.com/paper/2205.04952/) (2022)
- [Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation](https://mlsanity.com/paper/2202.13363/) (2022)
- [Enhancing Zero-Shot Many to Many Voice Conversion with Self-Attention VAE](https://mlsanity.com/paper/2203.16037/) (2022)
- [Causal Prototype-inspired Contrast Adaptation for Unsupervised Domain Adaptive Semantic Segmentation of High-resolution Remote Sensing Imagery](https://mlsanity.com/paper/2403.03704/) (2024)
- [MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech](https://mlsanity.com/paper/2410.03192/) (2024)
- [Beyond Voice Identity Conversion: Manipulating Voice Attributes by Adversarial Learning of Structured Disentangled Representations](https://mlsanity.com/paper/2107.12346/) (2021)
- [CycleFlow: Purify Information Factors by Cycle Loss](https://mlsanity.com/paper/2110.09928/) (2021)
- [Diversifying Joint Vision-Language Tokenization Learning](https://mlsanity.com/paper/2306.03421/) (2023)
- [Operational Latent Spaces](https://mlsanity.com/paper/2406.02699/) (2024)

## Abstract (arXiv)

Voice style transfer, also called voice conversion, seeks to modify one speaker's voice to generate speech as if it came from another (target) speaker. Previous works have made progress on voice conversion with parallel training data and pre-known speakers. However, zero-shot voice style transfer, which learns from non-parallel data and generates voices for previously unseen speakers, remains a challenging problem. We propose a novel zero-shot voice transfer method via disentangled representation learning. The proposed method first encodes speaker-related style and voice content of each input voice into separated low-dimensional embedding spaces, and then transfers to a new voice by combining the source content embedding and target style embedding through a decoder. With information-theoretic guidance, the style and content embedding spaces are representative and (ideally) independent of each other. On real-world VCTK datasets, our method outperforms other baselines and obtains state-of-the-art results in terms of transfer accuracy and voice naturalness for voice style transfer experiments under both many-to-many and zero-shot setups.