mlsanityData to 7 Oct 2026

Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation

arXiv 2311.04693 ↗First posted 8 Nov 2023eess.AS cs.AI cs.SDcited by 30

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Although voice conversion (VC) systems have shown a remarkable ability to transfer voice style, existing methods still have an inaccurate pitch and low speaker adaptation quality. To address these challenges, we introduce Diff-HierVC, a hierarchical VC system based on two diffusion models. We first introduce DiffPitch, which can effectively generate F0 with the target voice style. Subsequently, the generated F0 is fed to DiffVoice to convert the speech with a target voice style. Furthermore, using the source-filter encoder, we disentangle the speech and use the converted Mel-spectrogram as a data-driven prior in DiffVoice to improve the voice style transfer capacity. Finally, by using the masked prior in diffusion models, our model can improve the speaker adaptation quality. Experimental results verify the superiority of our model in pitch generation and voice style transfer performance, and our model also achieves a CER of 0.83% and EER of 3.29% in zero-shot VC scenarios.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 28 of its 35 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

28 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion25 May
  2. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation19 Dec
  3. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis21 Apr
  4. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain31 Mar
  5. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme28 Sep
  6. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 21

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2022High Fidelity Neural Audio Compressioncited by 673
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  8. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  12. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
9 more
  1. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 113
  2. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  3. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
  4. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  5. 2020F0-consistent many-to-many non-parallel voice conversion via conditional autoencodercited by 35
  6. 2020Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversioncited by 22
  7. 2020The ins and outs of speaker recognition: lessons from VoxSRC 2020cited by 21
  8. 2021Towards Natural and Controllable Cross-Lingual Voice Conversion Based on Neural TTS Model and Phonetic Posteriorgramcited by 3
  9. 2022DeID-VC: Speaker De-identification via Zero-shot Pseudo Voice Conversioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 15; 2026: 6.

30 papers.

Sort:

By area:

  1. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  2. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  3. 2024vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoderscited by 6
  4. 2024VoiceWukong: Benchmarking Deepfake Voice Detectioncited by 5
  5. 2025Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversioncited by 5
  6. 2024Diff-ETS: Learning a Diffusion Probabilistic Model for Electromyography-to-Speech Conversioncited by 4
  7. 2025TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detectioncited by 4
  8. 2025VoicePrompter: Robust Zero-Shot Voice Conversion with Voice Prompt and Conditional Flow Matchingcited by 3
  9. 2025Can Diffusion Models Disentangle? A Theoretical Perspectivecited by 3
  10. 2025DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detectioncited by 3
  11. 2026Diffusion Timbre Transfer Via Mutual Information Guided Inpaintingcited by 3
  12. 2024SingVisio: Visual Analytics of Diffusion Model for Singing Voice Conversioncited by 2
18 more
  1. 2024MulliVC: Multi-lingual Voice Conversion With Cycle Consistencycited by 2
  2. 2025EZ-VC: Easy Zero-shot Any-to-Any Voice Conversioncited by 2
  3. 2025UniCoM: A Universal Code-Switching Speech Generatorcited by 2
  4. 2025FasterVoiceGrad: Faster One-step Diffusion-Based Voice Conversion with Adversarial Diffusion Conversion Distillationcited by 2
  5. 2025LatentVoiceGrad: Nonparallel Voice Conversion with Latent Diffusion/Flow-Matching Modelscited by 2
  6. 2024Discrete Unit based Masking for Improving Disentanglement in Voice Conversioncited by 1
  7. 2024Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modelingcited by 1
  8. 2025When Humans Growl and Birds Speak: High-Fidelity Voice Conversion from Human to Animal and Designed Soundscited by 1
  9. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matchingcited by 1
  10. 2025CO-VADA: A Confidence-Oriented Voice Augmentation Debiasing Approach for Fair Speech Emotion Recognitioncited by 1
  11. 2025MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidancescited by 1
  12. 2026MeanVoiceFlow: One-step Nonparallel Voice Conversion with Mean Flowscited by 1
  13. 2026The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluationcited by 1
  14. 2025HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speechcited by 0
  15. 2025Bridging the Language Gap: Synthetic Voice Diversity via Latent Mixup for Equitable Speech Recognitioncited by 0
  16. 2026VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defensescited by 0
  17. 2026MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corporacited by 0
  18. 2026MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReFlow-VC: Zero-shot Voice Conversion Based on Rectified Flow and Speaker Feature Optimization
  2. 2025ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training
  3. 2024CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion
  4. 2024FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation
  5. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching
  6. 2025AdaptVC: High Quality Voice Conversion with Adaptive Learning
  7. 2025REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers
  8. 2021DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion