mlsanityData to 14 Oct 2026

RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding

arXiv 2506.10289 ↗First posted 12 Jun 2025eess.AS cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Voice conversion has emerged as a pivotal technology in numerous applications ranging from assistive communication to entertainment. In this paper, we present RT-VC, a zero-shot real-time voice conversion system that delivers ultra-low latency and high-quality performance. Our approach leverages an articulatory feature space to naturally disentangle content and speaker characteristics, facilitating more robust and interpretable voice transformations. Additionally, the integration of differentiable digital signal processing (DDSP) enables efficient vocoding directly from articulatory features, significantly reducing conversion latency. Experimental evaluations demonstrate that, while maintaining synthesis quality comparable to the current state-of-the-art (SOTA) method, RT-VC achieves a CPU latency of 61.4 ms, representing a 13.3% reduction in latency.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

36 of 41 references.

References · 36

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  8. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  9. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  10. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  11. 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 110
  12. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
24 more
  1. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 88
  2. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  3. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  4. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 80
  5. 2018StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networkscited by 78
  6. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  7. 2019CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversioncited by 46
  8. 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
  9. 2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 31
  10. 2023RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Musiccited by 30
  11. 2022Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversioncited by 28
  12. 2024Coding Speech through Vocal Tract Kinematicscited by 18
  13. 2023Self-Supervised Models of Speech Infer Universal Articulatory Kinematicscited by 16
  14. 2023Speaker-Independent Acoustic-to-Articulatory Speech Inversioncited by 15
  15. 2024StreamVC: Real-Time Low-Latency Voice Conversioncited by 15
  16. 2020Cotatron: Transcription-Guided Speech Encoder for Any-to-Many Voice Conversion without Parallel Datacited by 14
  17. 2022Deep Speech Synthesis from Articulatory Representationscited by 10
  18. 2023Improving Speech Inversion Through Self-Supervised Embeddings and Enhanced Tract Variablescited by 7
  19. 2021Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Taskscited by 5
  20. 2021EMA2S: An End-to-End Multimodal Articulatory-to-Speech Systemcited by 5
  21. 2022The Secret Source : Incorporating Source Features to Improve Acoustic-to-Articulatory Speech Inversioncited by 5
  22. 2023Style Modeling for Multi-Speaker Articulation-to-Speechcited by 4
  23. 2022HiFi-VC: High Quality ASR-Based Voice Conversioncited by 3
  24. 2024Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSPcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025HarmonicAttack: An Adaptive Cross-Domain Audio Watermark Removalcited by 3
  2. 2025Provable Speech Attributes Conversion via Latent Independencecited by 1
  3. 2026StyleStream: Real-Time Zero-Shot Voice Style Conversioncited by 0
  4. 2026VOSSA: Voiceprint Optimization for Streaming Speech Architecturescited by 0