mlsanityData to 7 Oct 2026

Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme

arXiv 2109.13821 ↗First posted 28 Sep 2021cs.SD cs.LG stat.MLcited by 99

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Voice conversion is a common speech synthesis task which can be solved in different ways depending on a particular real-world scenario. The most challenging one often referred to as one-shot many-to-many voice conversion consists in copying the target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We present a scalable high-quality solution based on diffusion probabilistic modeling and demonstrate its superior quality compared to state-of-the-art one-shot voice conversion approaches. Moreover, focusing on real-time applications, we investigate general principles which can make diffusion models faster while keeping synthesis quality at a high level. As a result, we develop a novel Stochastic Differential Equations solver suitable for various diffusion model types and generative tasks as shown through empirical studies and justify it by theoretical analysis.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 30 of its 46 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

30 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion28 May
  2. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  3. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
  4. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2021Variational Diffusion Modelscited by 1,145
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  12. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
14 more
  1. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  2. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  3. 2021Learning to Efficiently Sample from Diffusion Probabilistic Modelscited by 129
  4. 2021Noise Estimation for Generative Diffusion Modelscited by 94
  5. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  6. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  7. 2020AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalizationcited by 44
  8. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
  9. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  10. 2020F0-consistent many-to-many non-parallel voice conversion via conditional autoencodercited by 35
  11. 2020VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecturecited by 34
  12. 2020VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamicscited by 14
  13. 2021Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniquescited by 11
  14. 2021Many-to-Many Voice Conversion based Feature Disentanglement using Variational Autoencodercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 24; 2024: 34; 2025: 25; 2026: 8.

99 papers.

Sort:

By area:

  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  2. 2022A Survey on Generative Diffusion Modelcited by 238
  3. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  4. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  5. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  6. 2024ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scalecited by 88
  7. 2022Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Modelscited by 83
  8. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 81
  9. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  10. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matchingcited by 49
  11. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  12. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
38 more
  1. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 38
  2. 2023Diffusion Model-Augmented Behavioral Cloningcited by 37
  3. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  4. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Modelscited by 34
  5. 2023Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancementcited by 30
  6. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  7. 2023Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesiscited by 29
  8. 2021EdiTTS: Score-based Editing for Controllable Text-to-Speechcited by 28
  9. 2024FlashSpeech: Efficient Zero-Shot Speech Synthesiscited by 22
  10. 2024Consistency Diffusion Bridge Modelscited by 22
  11. 2023UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Datacited by 15
  12. 2024StreamVC: Real-Time Low-Latency Voice Conversioncited by 15
  13. 2023DPM-OT: A New Diffusion Probabilistic Model Based on Optimal Transportcited by 14
  14. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matchingcited by 14
  15. 2023LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speechcited by 13
  16. 2025TAMO: Fine-Grained Root Cause Analysis via Tool-Assisted LLM Agent with Multi-Modality Observation Data in Cloud-Native Systemscited by 13
  17. 2023DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extractioncited by 12
  18. 2023Towards More Accurate Diffusion Model Acceleration with A Timestep Tunercited by 12
  19. 2024I Can Hear You: Selective Robust Training for Deepfake Audio Detectioncited by 12
  20. 2025A Diffusion Model Translator for Efficient Image-to-Image Translationcited by 12
  21. 2022Learning the Beauty in Songs: Neural Singing Voice Beautifiercited by 10
  22. 2023HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfercited by 9
  23. 2024End-to-end streaming model for low-latency speech anonymizationcited by 9
  24. 2023EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Datacited by 7
  25. 2024VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editingcited by 7
  26. 2024Counterfactual Data Augmentation with Denoising Diffusion for Graph Anomaly Detectioncited by 7
  27. 2024FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillationcited by 7
  28. 2023DuTa-VC: A Duration-aware Typical-to-atypical Voice Conversion Approach with Diffusion Probabilistic Modelcited by 6
  29. 2024LDM-SVC: Latent Diffusion Model Based Zero-Shot Any-to-Any Singing Voice Conversion with Singer Guidancecited by 6
  30. 2024vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoderscited by 6
  31. 2025Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learningcited by 6
  32. 2025STOPA: A Database of Systematic VariaTion Of DeePfake Audio for Open-Set Source Tracing and Attributioncited by 6
  33. 2024Generative Modelling with High-Order Langevin Dynamicscited by 5
  34. 2024A Variance-Preserving Interpolation Approach for Diffusion Models with Applications to Single Channel Speech Enhancement and Recognitioncited by 5
  35. 2025Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversioncited by 5
  36. 2023Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakescited by 4
  37. 2024Optimal Transport Maps are Good Voice Converterscited by 4
  38. 2025ReFlow-VC: Zero-shot Voice Conversion Based on Rectified Flow and Speaker Feature Optimizationcited by 4

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
  2. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
  3. 2023Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion
  4. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  5. 2025Discrete-Time Diffusion-Like Models for Speech Synthesis
  6. 2023ReFlow-TTS: A Rectified Flow Model for High-fidelity Text-to-Speech
  7. 2024RDSinger: Reference-based Diffusion Network for Singing Voice Synthesis
  8. 2024CoMoSVC: Consistency Model-based Singing Voice Conversion