mlsanityData to 7 Oct 2026

Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversion

arXiv 2311.08104 ↗First posted 14 Nov 2023cs.SD cs.AI eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Research on deep learning-powered voice conversion (VC) in speech-to-speech scenarios is getting increasingly popular. Although many of the works in the field of voice conversion share a common global pipeline, there is a considerable diversity in the underlying structures, methods, and neural sub-blocks used across research efforts. Thus, obtaining a comprehensive understanding of the reasons behind the choice of the different methods in the voice conversion pipeline can be challenging, and the actual hurdles in the proposed solutions are often unclear. To shed light on these aspects, this paper presents a scoping review that explores the use of deep learning in speech analysis, synthesis, and disentangled speech representation learning within modern voice conversion systems. We screened 621 publications from more than 38 different venues between the years 2017 and 2023, followed by an in-depth review of a final database consisting of 123 eligible studies. Based on the review, we summarise the most frequently used approaches to voice conversion based on deep learning and highlight common pitfalls within the community. Lastly, we condense the knowledge gathered, identify main challenges and provide recommendations for future research directions.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 50 of its 100 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

50 of 100 references.

References · 50

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  7. 2018Efficient Neural Audio Synthesiscited by 218
  8. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  9. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  10. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
  11. 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 109
  12. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
38 more
  1. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  2. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  3. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  4. 2019CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversioncited by 45
  5. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38
  6. 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
  7. 2020F0-consistent many-to-many non-parallel voice conversion via conditional autoencodercited by 35
  8. 2020VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecturecited by 34
  9. 2022AVQVC: One-shot Voice Conversion by Vector Quantization with applying contrastive learningcited by 24
  10. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  11. 2020Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representationscited by 18
  12. 2023LM-VC: Zero-shot Voice Conversion via Speech Generation based on Language Modelscited by 18
  13. 2021Adversarially learning disentangled speech representations for robust multi-factor voice conversioncited by 14
  14. 2022Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversioncited by 14
  15. 2020Unsupervised Representation Disentanglement using Cross Domain Features and Adversarial Learning in Variational Autoencoder based Voice Conversioncited by 13
  16. 2021Disentanglement of Emotional Style and Speaker Identity for Expressive Voice Conversioncited by 13
  17. 2021Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniquescited by 11
  18. 2020GAZEV: GAN-Based Zero-Shot Voice Conversion over Non-parallel Speech Corpuscited by 10
  19. 2022Towards Improved Zero-shot Voice Conversion with Conditional DSVAEcited by 10
  20. 2021NVC-Net: End-to-End Adversarial Voice Conversioncited by 9
  21. 2022DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learningcited by 9
  22. 2020Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoderscited by 8
  23. 2021Timbre Transfer with Variational Auto Encoding and Cycle-Consistent Adversarial Networkscited by 8
  24. 2021An Improved StarGAN for Emotional Voice Conversion: Enhancing Voice Quality and Data Augmentationcited by 7
  25. 2021Training Robust Zero-Shot Voice Conversion Models with Self-supervised Featurescited by 7
  26. 2020Quasi-Periodic WaveNet: An Autoregressive Raw Waveform Generative Model with Pitch-dependent Dilated Convolution Neural Networkcited by 6
  27. 2021Zero-shot Voice Conversion via Self-supervised Prosody Representation Learningcited by 6
  28. 2022Streamable Neural Audio Synthesis With Non-Causal Convolutionscited by 6
  29. 2021SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machinescited by 5
  30. 2019Investigation of F0 conditioning and Fully Convolutional Networks in Variational Autoencoder based Voice Conversioncited by 4
  31. 2021StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contextscited by 4
  32. 2021Decoupling Speaker-Independent Emotions for Voice Conversion Via Source-Filter Networkscited by 4
  33. 2023DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Codingcited by 4
  34. 2021An Adaptive Learning based Generative Adversarial Network for One-To-One Voice Conversioncited by 3
  35. 2023DualVC 2: Dynamic Masked Convolution for Unified Streaming and Non-Streaming Voice Conversioncited by 3
  36. 2019DNN-based cross-lingual voice conversion using Bottleneck Featurescited by 2
  37. 2021DeepA: A Deep Neural Analyzer For Speech And Singing Vocodingcited by 1
  38. 2022End-to-End Voice Conversion with Information Perturbationcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Audio-based Kinship Verification Using Age Domain Conversioncited by 3
  2. 2025StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversioncited by 1
  3. 2025MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglementcited by 0