mlsanityData to 14 Oct 2026

LatentVoiceGrad: Nonparallel Voice Conversion with Latent Diffusion/Flow-Matching Models

arXiv 2509.08379 ↗First posted 10 Sep 2025cs.SD eess.AScited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Previously, we introduced VoiceGrad, a nonparallel voice conversion (VC) technique enabling mel-spectrogram conversion from source to target speakers using a score-based diffusion model. The concept involves training a score network to predict the gradient of the log density of mel-spectrograms from various speakers. VC is executed by iteratively adjusting an input mel-spectrogram until resembling the target speaker's. However, challenges persist: audio quality needs improvement, and conversion is slower compared to modern VC methods designed to operate at very high speeds. To address these, we introduce latent diffusion models into VoiceGrad, proposing an improved version with reverse diffusion in the autoencoder bottleneck. Additionally, we propose using a flow matching model as an alternative to the diffusion model to further speed up the conversion process without compromising the conversion quality. Experimental results show enhanced speech quality and accelerated conversion compared to the original.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 51 of its 72 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

51 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching6 Dec
  2. 2024FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation3 Sep
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  4. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation8 Nov
  5. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme28 Sep
  6. 2021DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion28 May

Other references · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2018Neural Ordinary Differential Equationscited by 3,779
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  10. 2016Density estimation using Real NVPcited by 2,042
  11. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
33 more
  1. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  2. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  3. 2016Least Squares Generative Adversarial Networkscited by 1,001
  4. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  6. 2014Semi-Supervised Learning with Deep Generative Modelscited by 496
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  9. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  10. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  11. 2017DualGAN: Unsupervised Dual Learning for Image-to-Image Translationcited by 310
  12. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  13. 2017Learning to Discover Cross-Domain Relations with Generative Adversarial Networkscited by 278
  14. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  15. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  16. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  17. 2018StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networkscited by 78
  18. 2022The VoiceMOS Challenge 2022cited by 57
  19. 2019CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversioncited by 46
  20. 2016Voice Conversion from Non-parallel Corpora Using Variational Auto-encodercited by 45
  21. 2020AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalizationcited by 44
  22. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
  23. 2017Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networkscited by 38
  24. 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
  25. 2019Evaluating Voice Conversion-based Privacy Protection against Informed Attackerscited by 31
  26. 2019Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversioncited by 29
  27. 2020CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversioncited by 25
  28. 2021MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Framescited by 18
  29. 2019Non-Parallel Voice Conversion with Cyclic Variational Autoencodercited by 15
  30. 2020VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamicscited by 14
  31. 2020Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networkscited by 6
  32. 2018Voice Conversion Based on Cross-Domain Features Using Variational Auto Encoderscited by 5
  33. 2024Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026MeanVoiceFlow: One-step Nonparallel Voice Conversion with Mean Flowscited by 1
  2. 2026MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FasterVoiceGrad: Faster One-step Diffusion-Based Voice Conversion with Adversarial Diffusion Conversion Distillation
  2. 2025ReFlow-VC: Zero-shot Voice Conversion Based on Rectified Flow and Speaker Feature Optimization
  3. 2024LCM-SVC: Latent Diffusion Model Based Singing Voice Conversion with Inference Acceleration via Latent Consistency Distillation
  4. 2026One-Step Voice Conversion by Learning kNN Transport in WavLM Space ↗
  5. 2024Zero-shot Voice Conversion with Diffusion Transformers
  6. 2025WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching
  7. 2021PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior
  8. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping