# vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders

arXiv: https://arxiv.org/abs/2409.01995 · HTML: https://mlsanity.com/paper/2409.01995/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 43 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 30 of 42)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [Voice Conversion With Just Nearest Neighbors](https://mlsanity.com/paper/2305.18975/) (2023)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)
- [NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis](https://mlsanity.com/paper/2211.09407/) (2022)
- [DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion](https://mlsanity.com/paper/2305.15816/) (2023)
- [UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding](https://mlsanity.com/paper/2306.07547/) (2023)
- [VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature](https://mlsanity.com/paper/2204.00768/) (2022)
- [Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation](https://mlsanity.com/paper/2311.04693/) (2023)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)

## Built on by (most cited first, 6 of 6)

- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion](https://mlsanity.com/paper/2505.16691/) (2025)
- [X-VC: Zero-shot Streaming Voice Conversion in Codec Space](https://mlsanity.com/paper/2604.12456/) (2026)
- [Benchmarking Prosody Encoding in Discrete Speech Tokens](https://mlsanity.com/paper/2508.11224/) (2025)
- [Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition](https://mlsanity.com/paper/2606.05678/) (2026)
- [OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL](https://mlsanity.com/paper/2606.30356/) (2026)

## Abstract (arXiv)

We propose a new speech discrete token vocoder, vec2wav 2.0, which advances voice conversion (VC). We use discrete tokens from speech self-supervised models as the content features of source speech, and treat VC as a prompted vocoding task. To amend the loss of speaker timbre in the content tokens, vec2wav 2.0 utilizes the WavLM features to provide strong timbre-dependent information. A novel adaptive Snake activation function is proposed to better incorporate timbre into the waveform reconstruction process. In this way, vec2wav 2.0 learns to alter the speaker timbre appropriately given different reference prompts. Also, no supervised data is required for vec2wav 2.0 to be effectively trained. Experimental results demonstrate that vec2wav 2.0 outperforms all other baselines to a considerable margin in terms of audio quality and speaker similarity in any-to-any VC. Ablation studies verify the effects made by the proposed techniques. Moreover, vec2wav 2.0 achieves competitive cross-lingual VC even only trained on monolingual corpus. Thus, vec2wav 2.0 shows timbre can potentially be manipulated only by speech token vocoders, pushing the frontiers of VC and speech synthesis.