mlsanityData to 7 Oct 2026

vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders

arXiv 2409.01995 ↗First posted 3 Sep 2024eess.AS cs.AI cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

We propose a new speech discrete token vocoder, vec2wav 2.0, which advances voice conversion (VC). We use discrete tokens from speech self-supervised models as the content features of source speech, and treat VC as a prompted vocoding task. To amend the loss of speaker timbre in the content tokens, vec2wav 2.0 utilizes the WavLM features to provide strong timbre-dependent information. A novel adaptive Snake activation function is proposed to better incorporate timbre into the waveform reconstruction process. In this way, vec2wav 2.0 learns to alter the speaker timbre appropriately given different reference prompts. Also, no supervised data is required for vec2wav 2.0 to be effectively trained. Experimental results demonstrate that vec2wav 2.0 outperforms all other baselines to a considerable margin in terms of audio quality and speaker similarity in any-to-any VC. Ablation studies verify the effects made by the proposed techniques. Moreover, vec2wav 2.0 achieves competitive cross-lingual VC even only trained on monolingual corpus. Thus, vec2wav 2.0 shows timbre can potentially be manipulated only by speech token vocoders, pushing the frontiers of VC and speech synthesis.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 42 of its 43 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

42 of 43 references.

References · 42

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  6. 2022High Fidelity Neural Audio Compressioncited by 673
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  10. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  12. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
30 more
  1. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  2. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  3. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  4. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  5. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  6. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Schemecited by 99
  7. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 97
  8. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
  9. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
  10. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  11. 2023Voice Conversion With Just Nearest Neighborscited by 67
  12. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  13. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38
  14. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 38
  15. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  16. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  17. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  18. 2022SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottleneckscited by 27
  19. 2023Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTScited by 24
  20. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 22
  21. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  22. 2023SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attentioncited by 19
  23. 2023ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representationscited by 15
  24. 2022Text-free non-parallel many-to-many voice conversion using normalising flowscited by 10
  25. 2024Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representationscited by 7
  26. 2023DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speechcited by 5
  27. 2023SelfVC: Voice Conversion With Iterative Refinement using Self Transformationscited by 5
  28. 2024Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategycited by 4
  29. 2022A unified one-shot prosody and speaker conversion system with self-supervised discrete speech unitscited by 3
  30. 2023Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challengecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2025EZ-VC: Easy Zero-shot Any-to-Any Voice Conversioncited by 2
  3. 2026X-VC: Zero-shot Streaming Voice Conversion in Codec Spacecited by 2
  4. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  5. 2026Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognitioncited by 0
  6. 2026OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSLcited by 0