mlsanityData to 14 Oct 2026

Zero-shot Voice Conversion with Diffusion Transformers

arXiv 2411.09943 ↗First posted 15 Nov 2024cs.SD cs.LG eess.AScited by 43

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Zero-shot voice conversion aims to transform a source speech utterance to match the timbre of a reference speech from an unseen speaker. Traditional approaches struggle with timbre leakage, insufficient timbre representation, and mismatches between training and inference tasks. We propose Seed-VC, a novel framework that addresses these issues by introducing an external timbre shifter during training to perturb the source speech timbre, mitigating leakage and aligning training with inference. Additionally, we employ a diffusion transformer that leverages the entire reference speech context, capturing fine-grained timbre features through in-context learning. Experiments demonstrate that Seed-VC outperforms strong baselines like OpenVoice and CosyVoice, achieving higher speaker similarity and lower word error rates in zero-shot voice conversion tasks. We further extend our approach to zero-shot singing voice conversion by incorporating fundamental frequency (F0) conditioning, resulting in comparative performance to current state-of-the-art methods. Our findings highlight the effectiveness of Seed-VC in overcoming core challenges, paving the way for more accurate and versatile voice conversion systems.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 39 of its 42 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

39 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  2. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors17 Jun
  3. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation8 Nov
  4. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion25 May
  5. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  6. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme28 Sep
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  12. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
20 more
  1. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  2. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  3. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  4. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  5. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  6. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  7. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  8. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  9. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  10. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  11. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  12. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  13. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 102
  14. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 80
  15. 2023Voice Conversion With Just Nearest Neighborscited by 68
  16. 2023OpenVoice: Versatile Instant Voice Cloningcited by 32
  17. 2023RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Musiccited by 30
  18. 2024StreamVC: Real-Time Low-Latency Voice Conversioncited by 15
  19. 2020PPG-based singing voice conversion with adversarial representation learningcited by 12
  20. 2024MulliVC: Multi-lingual Voice Conversion With Cycle Consistencycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 29.

43 papers.

Sort:

By area:

  1. 2025Kimi-Audio Technical Reportcited by 222
  2. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  3. 2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
  4. 2025SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understandingcited by 6
  5. 2026IndexTTS 2.5 Technical Reportcited by 6
  6. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  7. 2025TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detectioncited by 4
  8. 2025REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformerscited by 3
  9. 2025YingMusic-SVC: Real-World Robust Zero-Shot Singing Voice Conversion with Flow-GRPO and Singing-Specific Inductive Biasescited by 3
  10. 2026AT-ADD: All-Type Audio Deepfake Detection Challenge Evaluation Plancited by 3
  11. 2025EZ-VC: Easy Zero-shot Any-to-Any Voice Conversioncited by 2
  12. 2025De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attackscited by 2
31 more
  1. 2025UniCoM: A Universal Code-Switching Speech Generatorcited by 2
  2. 2025Entropy-based Coarse and Compressed Semantic Speech Representation Learningcited by 2
  3. 2026Token Pruning for In-Context Generation in Diffusion Transformerscited by 2
  4. 2026X-VC: Zero-shot Streaming Voice Conversion in Codec Spacecited by 2
  5. 2026How Far Are Video Models from True Multimodal Reasoning?cited by 2
  6. 2025R2-SVC: Towards Real-World Robust and Expressive Zero-shot Singing Voice Conversioncited by 1
  7. 2026PHONOS: PHOnetic Neutralization for Online Streaming Applicationscited by 1
  8. 2026MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corporacited by 1
  9. 2026From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interactioncited by 1
  10. 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editingcited by 1
  11. 2025SAGE-LD: Towards Scalable and Generalizable End-to-End Language Diarization via Simulated Data Augmentationcited by 0
  12. 2026QASA: Quality-Aware Semantic Augmentation for Robust Multimodal Sentiment Analysiscited by 0
  13. 2026Controllable Dysarthric Speech Synthesis with Patient-Specific Conditioning for Speaker-Diverse ASR Augmentationcited by 0
  14. 2026StyleStream: Real-Time Zero-Shot Voice Style Conversioncited by 0
  15. 2026From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Datacited by 0
  16. 2026Vibrato Expression Control for Singing Voice Conversion with Improving Independent Controlcited by 0
  17. 2026Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approachcited by 0
  18. 2026Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesiscited by 0
  19. 2026GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speechcited by 0
  20. 2026VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversioncited by 0
  21. 2026MADBench: A Benchmark for Modality-Aware Audio Deepfake Detectioncited by 0
  22. 2026Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encodercited by 0
  23. 2026Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenizationcited by 0
  24. 2026UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videoscited by 0
  25. 2026Deterministic Prompting for Speaker-Stable Low-Resource Greek TTScited by 0
  26. 2026COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoningcited by 0
  27. 2026Understanding Hyperspherical Geometry of ECAPA-TDNN Embedding and Its Impact on Zero-Shot Voice Conversioncited by 0
  28. 2026Personalized Automatic Speech Recognition for a Dysarthric and Tracheostomic Speaker using Artificial Conversationscited by 0
  29. 2026Exploiting Acoustic and Content-Oriented Speaker Verification Attacks Against Multilingual Voice Anonymizationcited by 0
  30. 2026AegisVox: Factorized Privacy and Clinical Preservation for Long-Form Speechcited by 0
  31. 2026AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
  2. 2024CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion
  3. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching
  4. 2025MeanVC: Lightweight and Streaming Zero-Shot Voice Conversion via Mean Flows
  5. 2024Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling
  6. 2024Zero-Shot Voice Conversion via Content-Aware Timbre Ensemble and Conditional Flow Matching
  7. 2025Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion
  8. 2025ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training