mlsanityData to 14 Oct 2026

REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers

arXiv 2508.04996 ↗First posted 7 Aug 2025eess.AScited by 3

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In real-world voice conversion applications, environmental noise in source speech and user demands for expressive output pose critical challenges. Traditional ASR-based methods ensure noise robustness but suppress prosody richness, while SSL-based models improve expressiveness but suffer from timbre leakage and noise sensitivity. This paper proposes REF-VC, a noise-robust expressive voice conversion system. Key innovations include: (1) A random erasing strategy to mitigate the information redundancy inherent in SSL features, enhancing noise robustness and expressiveness; (2) Implicit alignment inspired by E2TTS to suppress non-essential feature reconstruction; (3) Integration of Shortcut Models to accelerate flow matching inference, significantly reducing to 4 steps. Experimental results demonstrate that REF-VC outperforms baselines such as Seed-VC in zero-shot scenarios on the noisy set, while also performing comparably to Seed-VC on the clean set. In addition, REF-VC can be compatible with singing voice conversion within one model.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 23 of its 32 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

23 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Zero-shot Voice Conversion with Diffusion Transformers15 Nov
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct

Other references · 21

  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  6. 2024One Step Diffusion via Shortcut Modelscited by 391
  7. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  8. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  9. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  10. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 80
  11. 2022WeNet 2.0: More Productive End-to-End Speech Recognition Toolkitcited by 47
  12. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
9 more
  1. 2021S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representationscited by 17
  2. 2020Accent and Speaker Disentanglement in Many-to-many Voice Conversioncited by 15
  3. 2023Self-Supervised Representations for Singing Voice Conversioncited by 8
  4. 2023VITS-Based Singing Voice Conversion Leveraging Whisper and multi-scale F0 Modelingcited by 6
  5. 2022Noise-robust voice conversion with domain adversarial trainingcited by 5
  6. 2023SelfVC: Voice Conversion With Iterative Refinement using Self Transformationscited by 5
  7. 2023DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Codingcited by 4
  8. 2024Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategycited by 4
  9. 2021Toward Degradation-Robust Voice Conversioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025R2-SVC: Towards Real-World Robust and Expressive Zero-shot Singing Voice Conversioncited by 1
  2. 2026UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videoscited by 0
  3. 2026AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReFlow-VC: Zero-shot Voice Conversion Based on Rectified Flow and Speaker Feature Optimization
  2. 2025YingMusic-SVC: Real-World Robust Zero-Shot Singing Voice Conversion with Flow-GRPO and Singing-Specific Inductive Biases
  3. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
  4. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching
  5. 2021DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion
  6. 2021NoiseVC: Towards High Quality Zero-Shot Voice Conversion
  7. 2026MeanVoiceFlow: One-step Nonparallel Voice Conversion with Mean Flows
  8. 2025Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model