mlsanityData to 14 Oct 2026

A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion

arXiv 2111.02392 ↗First posted 3 Nov 2021eess.AS cs.SDcited by 61

Not on the map. Why?

Abstract (arXiv)

The goal of voice conversion is to transform source speech into a target voice, keeping the content unchanged. In this paper, we focus on self-supervised representation learning for voice conversion. Specifically, we compare discrete and soft speech units as input features. We find that discrete representations effectively remove speaker information but discard some linguistic content - leading to mispronunciations. As a solution, we propose soft speech units. To learn soft units, we predict a distribution over discrete speech units. By modeling uncertainty, soft units capture more content information, improving the intelligibility and naturalness of converted speech. Samples available at https://ubisoft-laforge.github.io/speech/soft-vc/. Code available at https://github.com/bshall/soft-vc/.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Nov 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 16; 2024: 12; 2025: 24; 2026: 7.

61 papers.

Sort:

By area:

  1. 2023Voice Conversion With Just Nearest Neighborscited by 68
  2. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  3. 2023OpenVoice: Versatile Instant Voice Cloningcited by 32
  4. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  5. 2023The Singing Voice Conversion Challenge 2023cited by 25
  6. 2023Multi-modal Facial Affective Analysis based on Masked Autoencodercited by 23
  7. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  8. 2024StreamVC: Real-Time Low-Latency Voice Conversioncited by 15
  9. 2023FSD: An Initial Chinese Dataset for Fake Song Detectioncited by 12
  10. 2022Self-Supervised Learning for Speech Enhancement through Synthesiscited by 10
  11. 2024End-to-end streaming model for low-latency speech anonymizationcited by 9
  12. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
38 more
  1. 2026AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoderscited by 8
  2. 2024A Benchmark for Multi-speaker Anonymizationcited by 7
  3. 2024Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programmingcited by 7
  4. 2023Rhythm Modeling for Voice Conversioncited by 6
  5. 2023Towards General-Purpose Text-Instruction-Guided Voice Conversioncited by 6
  6. 2024vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoderscited by 6
  7. 2023Zero-Shot Automatic Pronunciation Assessmentcited by 5
  8. 2026TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymizationcited by 5
  9. 2023High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Unitscited by 4
  10. 2023Disentanglement in a GAN for Unconditional Speech Synthesiscited by 4
  11. 2023Voice Conversion for Stuttered Speech, Instruments, Unseen Languages and Textually Described Voicescited by 4
  12. 2024Hear Your Face: Face-based voice conversion with F0 estimationcited by 4
  13. 2025ReFlow-VC: Zero-shot Voice Conversion Based on Rectified Flow and Speaker Feature Optimizationcited by 4
  14. 2025RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Codingcited by 4
  15. 2025Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversioncited by 4
  16. 2025Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?cited by 4
  17. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  18. 2025Unsupervised lexicon learning from speech is limited by representations rather than clusteringcited by 4
  19. 2023Low-latency Real-time Voice Conversion on CPUcited by 3
  20. 2024Disentangling segmental and prosodic factors to non-native speech comprehensibilitycited by 3
  21. 2025AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencodercited by 3
  22. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  23. 2025EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameterscited by 3
  24. 2025LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialectcited by 3
  25. 2025Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equationcited by 3
  26. 2025LinearVC: Linear transformations of self-supervised features through the lens of voice conversioncited by 3
  27. 2025Analyzing and Improving Speaker Similarity Assessment for Speech Synthesiscited by 3
  28. 2023A Comparative Study of Voice Conversion Models with Large-Scale Speech and Singing Data: The T13 Systems for the Singing Voice Conversion Challenge 2023cited by 2
  29. 2025Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversioncited by 2
  30. 2022A Temporal Extension of Latent Dirichlet Allocation for Unsupervised Acoustic Unit Discoverycited by 1
  31. 2023Adversarial Speaker Disentanglement Using Unannotated External Data for Self-supervised Representation Based Voice Conversioncited by 1
  32. 2023Vocoder drift compensation by x-vector alignment in speaker anonymisationcited by 1
  33. 2024Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Modelcited by 1
  34. 2024Discrete Unit based Masking for Improving Disentanglement in Voice Conversioncited by 1
  35. 2024Analytic Study of Text-Free Speech Synthesis for Raw Audio using a Self-Supervised Learning Modelcited by 1
  36. 2025DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibilitycited by 1
  37. 2025kNN-SVC: Robust Zero-Shot Singing Voice Conversion with Additive Synthesis and Concatenation Smoothness Optimizationcited by 1
  38. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matchingcited by 1

50 most cited of 61 All 61 (JSON)