# Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations

arXiv: https://arxiv.org/abs/1804.02812 · HTML: https://mlsanity.com/paper/1804.02812/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2018.

## Built on by (most cited first, 25 of 25)

- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation](https://mlsanity.com/paper/2104.07995/) (2021)
- [CDPAM: Contrastive learning for perceptual audio similarity](https://mlsanity.com/paper/2102.05109/) (2021)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [Adversarially learning disentangled speech representations for robust multi-factor voice conversion](https://mlsanity.com/paper/2102.00184/) (2021)
- [Disentangling Voice and Content with Self-Supervision for Speaker Recognition](https://mlsanity.com/paper/2310.01128/) (2023)
- [Many-to-Many Voice Conversion based Feature Disentanglement using Variational Autoencoder](https://mlsanity.com/paper/2107.06642/) (2021)
- [A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion](https://mlsanity.com/paper/2106.01415/) (2021)
- [Global Rhythm Style Transfer Without Text Transcriptions](https://mlsanity.com/paper/2106.08519/) (2021)
- [DisUnknown: Distilling Unknown Factors for Disentanglement Learning](https://mlsanity.com/paper/2109.08090/) (2021)
- [Disentangling speech from surroundings with neural embeddings](https://mlsanity.com/paper/2203.15578/) (2022)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)
- [Disentangled Representation Learning for RF Fingerprint Extraction under Unknown Channel Statistics](https://mlsanity.com/paper/2208.02724/) (2022)
- [Toward Degradation-Robust Voice Conversion](https://mlsanity.com/paper/2110.07537/) (2021)
- [Speak Like a Dog: Human to Non-human creature Voice Conversion](https://mlsanity.com/paper/2206.04780/) (2022)
- [Learning Robust Latent Representations for Controllable Speech Synthesis](https://mlsanity.com/paper/2105.04458/) (2021)
- [Conditional Deep Hierarchical Variational Autoencoder for Voice Conversion](https://mlsanity.com/paper/2112.02796/) (2021)
- [A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversion](https://mlsanity.com/paper/2302.13678/) (2023)
- [Cross-modal Face- and Voice-style Transfer](https://mlsanity.com/paper/2302.13838/) (2023)
- [Learning Disentangled Speech Representations](https://mlsanity.com/paper/2311.03389/) (2023)
- [MoEVC: A Mixture-of-experts Voice Conversion System with Sparse Gating Mechanism for Accelerating Online Computation](https://mlsanity.com/paper/1912.11984/) (2019)
- [CycleFlow: Purify Information Factors by Cycle Loss](https://mlsanity.com/paper/2110.09928/) (2021)
- [MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement](https://mlsanity.com/paper/2511.12074/) (2025)

## Abstract (arXiv)

Recently, cycle-consistent adversarial network (Cycle-GAN) has been successfully applied to voice conversion to a different speaker without parallel data, although in those approaches an individual model is needed for each target speaker. In this paper, we propose an adversarial learning framework for voice conversion, with which a single model can be trained to convert the voice to many different speakers, all without parallel data, by separating the speaker characteristics from the linguistic content in speech signals. An autoencoder is first trained to extract speaker-independent latent representations and speaker embedding separately using another auxiliary speaker classifier to regularize the latent representation. The decoder then takes the speaker-independent latent representation and the target speaker embedding as the input to generate the voice of the target speaker with the linguistic content of the source utterance. The quality of decoder output is further improved by patching with the residual signal produced by another pair of generator and discriminator. A target speaker set size of 20 was tested in the preliminary experiments, and very good voice quality was obtained. Conventional voice conversion metrics are reported. We also show that the speaker information has been properly reduced from the latent representations.