# Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion

arXiv: https://arxiv.org/abs/2505.24291 · HTML: https://mlsanity.com/paper/2505.24291/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 33 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 30 of 31)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer](https://mlsanity.com/paper/2409.00750/) (2024)
- [E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS](https://mlsanity.com/paper/2406.18009/) (2024)
- [SoundStorm: Efficient Parallel Audio Generation](https://mlsanity.com/paper/2305.09636/) (2023)
- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [Addressing Representation Collapse in Vector Quantized Models with One Linear Layer](https://mlsanity.com/paper/2411.02038/) (2024)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias](https://mlsanity.com/paper/2306.03509/) (2023)
- [Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2307.07218/) (2023)
- [DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion](https://mlsanity.com/paper/2305.15816/) (2023)
- [Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement](https://mlsanity.com/paper/2502.07243/) (2025)
- [Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation](https://mlsanity.com/paper/2311.04693/) (2023)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation](https://mlsanity.com/paper/2406.08203/) (2024)
- [ReFlow-TTS: A Rectified Flow Model for High-fidelity Text-to-Speech](https://mlsanity.com/paper/2309.17056/) (2023)
- [SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention](https://mlsanity.com/paper/2312.08676/) (2023)
- [ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations](https://mlsanity.com/paper/2302.08137/) (2023)
- [StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching](https://mlsanity.com/paper/2412.04724/) (2024)
- [Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model](https://mlsanity.com/paper/2502.05471/) (2025)
- [Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy](https://mlsanity.com/paper/2406.09844/) (2024)

## Built on by (most cited first, 5 of 5)

- [UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models](https://mlsanity.com/paper/2510.04593/) (2025)
- [Provable Speech Attributes Conversion via Latent Independence](https://mlsanity.com/paper/2510.05191/) (2025)
- [DeepASMR: LLM-Based Zero-Shot ASMR Speech Generation for Anyone of Any Voice](https://mlsanity.com/paper/2601.15596/) (2026)
- [Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems](https://mlsanity.com/paper/2605.09259/) (2026)
- [PC-Mix: Partial-Component Audio Spoofing Detection under Mixed Speech and Environmental Sound Conditions](https://mlsanity.com/paper/2607.10345/) (2026)

## Abstract (arXiv)

Currently, zero-shot voice conversion systems are capable of synthesizing the voice of unseen speakers. However, most existing approaches struggle to accurately replicate the speaking style of the source speaker or mimic the distinctive speaking style of the target speaker, thereby limiting the controllability of voice conversion. In this work, we propose Discl-VC, a novel voice conversion framework that disentangles content and prosody information from self-supervised speech representations and synthesizes the target speaker's voice through in-context learning with a flow matching transformer. To enable precise control over the prosody of generated speech, we introduce a mask generative transformer that predicts discrete prosody tokens in a non-autoregressive manner based on prompts. Experimental results demonstrate the superior performance of Discl-VC in zero-shot voice conversion and its remarkable accuracy in prosody control for synthesized speech.