mlsanityData to 13 Oct 2026

StreamVC: Real-Time Low-Latency Voice Conversion

arXiv 2401.03078 ↗First posted 5 Jan 2024eess.AS cs.LG cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

We present StreamVC, a streaming voice conversion solution that preserves the content and prosody of any source speech while matching the voice timbre from any target speech. Unlike previous approaches, StreamVC produces the resulting waveform at low latency from the input signal even on a mobile platform, making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. Our design leverages the architecture and training strategy of the SoundStream neural audio codec for lightweight high-quality speech synthesis. We demonstrate the feasibility of learning soft speech units causally, as well as the effectiveness of supplying whitened fundamental frequency information to improve pitch stability without leaking the source timbre information.

Key facts. Cited by 15 papers in the map (data through 13 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

21 of 26 references.

References · 21

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,306
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,625
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 762
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 643
  7. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 559
  8. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 353
  9. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 239
  10. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 190
  11. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Schemecited by 99
  12. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
9 more
  1. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
  2. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  3. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  4. 2020Streaming keyword spotting on mobile devicescited by 47
  5. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
  6. 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
  7. 2020Real-time Speech Frequency Bandwidth Extensioncited by 22
  8. 2023QuickVC: Any-to-many Voice Conversion Using Inverse Short-time Fourier Transform for Faster Conversioncited by 6
  9. 2022Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidancecited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 41
  2. 2024Coding Speech through Vocal Tract Kinematicscited by 18
  3. 2024HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAscited by 9
  4. 2025da4ml: Distributed Arithmetic for Real-time Neural Networks on FPGAscited by 8
  5. 2025RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Codingcited by 4
  6. 2025Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversioncited by 4
  7. 2025LinearVC: Linear transformations of self-supervised features through the lens of voice conversioncited by 3
  8. 2025PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objectivecited by 3
  9. 2025HarmonicAttack: An Adaptive Cross-Domain Audio Watermark Removalcited by 3
  10. 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
  11. 2025A stylometric analysis of speaker attribution from speech transcriptscited by 2
  12. 2026X-VC: Zero-shot Streaming Voice Conversion in Codec Spacecited by 2
3 more
  1. 2025Streaming Non-Autoregressive Model for Accent Conversion and Pronunciation Improvementcited by 1
  2. 2026Lightweight and perceptually-guided voice conversion for electro-laryngeal speechcited by 0
  3. 2026StyleStream: Real-Time Zero-Shot Voice Style Conversioncited by 0