# F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder

arXiv: https://arxiv.org/abs/2004.07370 · HTML: https://mlsanity.com/paper/2004.07370/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2020.

## Built on by (most cited first, 30 of 35)

- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion](https://mlsanity.com/paper/2305.15816/) (2023)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation](https://mlsanity.com/paper/2311.04693/) (2023)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion](https://mlsanity.com/paper/2208.08757/) (2022)
- [Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques](https://mlsanity.com/paper/2104.00931/) (2021)
- [Are disentangled representations all you need to build speaker anonymization systems?](https://mlsanity.com/paper/2208.10497/) (2022)
- [Cross-speaker style transfer for text-to-speech using data augmentation](https://mlsanity.com/paper/2202.05083/) (2022)
- [Audio Deepfake Detection Based on a Combination of F0 Information and Real Plus Imaginary Spectrogram Features](https://mlsanity.com/paper/2208.01214/) (2022)
- [HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer](https://mlsanity.com/paper/2307.16171/) (2023)
- [Many-to-Many Voice Conversion based Feature Disentanglement using Variational Autoencoder](https://mlsanity.com/paper/2107.06642/) (2021)
- [An Improved StarGAN for Emotional Voice Conversion: Enhancing Voice Quality and Data Augmentation](https://mlsanity.com/paper/2107.08361/) (2021)
- [Deepfake CAPTCHA: A Method for Preventing Fake Calls](https://mlsanity.com/paper/2301.03064/) (2023)
- [Global Rhythm Style Transfer Without Text Transcriptions](https://mlsanity.com/paper/2106.08519/) (2021)
- [Zero-shot Voice Conversion via Self-supervised Prosody Representation Learning](https://mlsanity.com/paper/2110.14422/) (2021)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)
- [SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement](https://mlsanity.com/paper/2407.07728/) (2024)
- [Speak Like a Dog: Human to Non-human creature Voice Conversion](https://mlsanity.com/paper/2206.04780/) (2022)
- [A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units](https://mlsanity.com/paper/2211.06535/) (2022)
- [Low-latency Real-time Voice Conversion on CPU](https://mlsanity.com/paper/2311.00873/) (2023)
- [Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversion](https://mlsanity.com/paper/2311.08104/) (2023)
- [Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints](https://mlsanity.com/paper/2407.03824/) (2024)
- [Generative Adversarial Network based Voice Conversion: Techniques, Challenges, and Recent Advancements](https://mlsanity.com/paper/2504.19197/) (2025)
- [A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversion](https://mlsanity.com/paper/2302.13678/) (2023)
- [Evaluation of Speaker Anonymization on Emotional Speech](https://mlsanity.com/paper/2305.01759/) (2023)

## Abstract (arXiv)

Non-parallel many-to-many voice conversion remains an interesting but challenging speech processing task. Many style-transfer-inspired methods such as generative adversarial networks (GANs) and variational autoencoders (VAEs) have been proposed. Recently, AutoVC, a conditional autoencoders (CAEs) based method achieved state-of-the-art results by disentangling the speaker identity and speech content using information-constraining bottlenecks, and it achieves zero-shot conversion by swapping in a different speaker's identity embedding to synthesize a new voice. However, we found that while speaker identity is disentangled from speech content, a significant amount of prosodic information, such as source F0, leaks through the bottleneck, causing target F0 to fluctuate unnaturally. Furthermore, AutoVC has no control of the converted F0 and thus unsuitable for many applications. In the paper, we modified and improved autoencoder-based voice conversion to disentangle content, F0, and speaker identity at the same time. Therefore, we can control the F0 contour, generate speech with F0 consistent with the target speaker, and significantly improve quality and similarity. We support our improvement through quantitative and qualitative analysis.