# An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning

arXiv: https://arxiv.org/abs/2008.03648 · HTML: https://mlsanity.com/paper/2008.03648/

## Key facts

- Cited by 112 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2020.

## Built on by (most cited first, 30 of 112)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [Voice Conversion With Just Nearest Neighbors](https://mlsanity.com/paper/2305.18975/) (2023)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversion](https://mlsanity.com/paper/2203.16705/) (2022)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [The Singing Voice Conversion Challenge 2023](https://mlsanity.com/paper/2306.14422/) (2023)
- [SafeEar: Content Privacy-Preserving Audio Deepfake Detection](https://mlsanity.com/paper/2409.09272/) (2024)
- [Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models](https://mlsanity.com/paper/2509.02859/) (2025)
- [WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration](https://mlsanity.com/paper/2210.01029/) (2022)
- [Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability](https://mlsanity.com/paper/2104.01408/) (2021)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts](https://mlsanity.com/paper/2309.09262/) (2023)
- [Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training](https://mlsanity.com/paper/2103.16809/) (2021)
- [CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing](https://mlsanity.com/paper/2202.09950/) (2022)
- [Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion](https://mlsanity.com/paper/2208.08757/) (2022)
- [VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over](https://mlsanity.com/paper/2110.03342/) (2021)
- [More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech](https://mlsanity.com/paper/2111.10139/) (2021)
- [Towards Improved Zero-shot Voice Conversion with Conditional DSVAE](https://mlsanity.com/paper/2205.05227/) (2022)
- [Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion](https://mlsanity.com/paper/2306.15875/) (2023)
- [Single and Multi-Speaker Cloned Voice Detection: From Perceptual to Learned Features](https://mlsanity.com/paper/2307.07683/) (2023)
- [NVC-Net: End-to-End Adversarial Voice Conversion](https://mlsanity.com/paper/2106.00992/) (2021)
- [Contrastive Fine-grained Class Clustering via Generative Adversarial Networks](https://mlsanity.com/paper/2112.14971/) (2021)
- [A Comparative Study of Self-supervised Speech Representation Based Voice Conversion](https://mlsanity.com/paper/2207.04356/) (2022)
- [Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignment](https://mlsanity.com/paper/2309.09470/) (2023)
- [GenVC: Self-Supervised Zero-Shot Voice Conversion](https://mlsanity.com/paper/2502.04519/) (2025)
- [Dispersed Pixel Perturbation-based Imperceptible Backdoor Trigger for Image Classifier Models](https://mlsanity.com/paper/2208.09336/) (2022)
- [Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?](https://mlsanity.com/paper/2311.07564/) (2023)

## Abstract (arXiv)

Speaker identity is one of the important characteristics of human speech. In voice conversion, we change the speaker identity from one to another, while keeping the linguistic content unchanged. Voice conversion involves multiple speech processing techniques, such as speech analysis, spectral conversion, prosody conversion, speaker characterization, and vocoding. With the recent advances in theory and practice, we are now able to produce human-like voice quality with high speaker similarity. In this paper, we provide a comprehensive overview of the state-of-the-art of voice conversion techniques and their performance evaluation methods from the statistical approaches to deep learning, and discuss their promise and limitations. We will also report the recent Voice Conversion Challenges (VCC), the performance of the current state of technology, and provide a summary of the available resources for voice conversion research.