# Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

arXiv: https://arxiv.org/abs/2502.07243 · HTML: https://mlsanity.com/paper/2502.07243/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2025.

## Built on by (most cited first, 30 of 35)

- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling](https://mlsanity.com/paper/2508.16790/) (2025)
- [CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering](https://mlsanity.com/paper/2602.03420/) (2026)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [IndexTTS 2.5 Technical Report](https://mlsanity.com/paper/2601.03888/) (2026)
- [Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion](https://mlsanity.com/paper/2505.24291/) (2025)
- [Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation](https://mlsanity.com/paper/2508.16332/) (2025)
- [YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance](https://mlsanity.com/paper/2512.04779/) (2025)
- [DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection](https://mlsanity.com/paper/2510.22622/) (2025)
- [AnyAccomp: Generalizable Accompaniment Generation via Quantized Melodic Bottleneck](https://mlsanity.com/paper/2509.14052/) (2025)
- [Rethinking Discrete Speech Representation Tokens for Accent Generation](https://mlsanity.com/paper/2601.19786/) (2026)
- [Scaling Speech Tokenizers with Diffusion Autoencoders](https://mlsanity.com/paper/2602.06602/) (2026)
- [Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec](https://mlsanity.com/paper/2603.05887/) (2026)
- [MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidances](https://mlsanity.com/paper/2509.17143/) (2025)
- [Provable Speech Attributes Conversion via Latent Independence](https://mlsanity.com/paper/2510.05191/) (2025)
- [U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation](https://mlsanity.com/paper/2510.16718/) (2025)
- [CodecFlow: Efficient Bandwidth Extension via Conditional Flow Matching in Neural Codec Latent Space](https://mlsanity.com/paper/2603.02022/) (2026)
- [PHONOS: PHOnetic Neutralization for Online Streaming Applications](https://mlsanity.com/paper/2603.27001/) (2026)
- [An Evaluation Framework for Text-to-Speech Voice Reconstruction](https://mlsanity.com/paper/2606.21343/) (2026)
- [Mitigating Intra-Speaker Variability in Diarization with Style-Controllable Speech Augmentation](https://mlsanity.com/paper/2509.14632/) (2025)
- [VoxMorph: Scalable Zero-shot Voice Identity Morphing via Disentangled Embeddings](https://mlsanity.com/paper/2601.20883/) (2026)
- [Tutti: Expressive Multi-Singer Synthesis via Structure-Level Timbre Control and Vocal Texture Modeling](https://mlsanity.com/paper/2602.08233/) (2026)
- [StyleStream: Real-Time Zero-Shot Voice Style Conversion](https://mlsanity.com/paper/2602.20113/) (2026)
- [Controllable Accent Normalization via Discrete Diffusion](https://mlsanity.com/paper/2603.14275/) (2026)
- [On the Emotion Understanding of Synthesized Speech](https://mlsanity.com/paper/2603.16483/) (2026)
- [Controllable Singing Style Conversion with Boundary-Aware Information Bottleneck](https://mlsanity.com/paper/2604.05526/) (2026)
- [From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data](https://mlsanity.com/paper/2606.08843/) (2026)
- [VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation](https://mlsanity.com/paper/2606.26534/) (2026)
- [TokAN: Accent Normalization Using Self-Supervised Speech Tokens](https://mlsanity.com/paper/2607.03928/) (2026)
- [NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization](https://mlsanity.com/paper/2607.03985/) (2026)

## Abstract (arXiv)

The imitation of voice, targeted on specific speech attributes such as timbre and speaking style, is crucial in speech generation. However, existing methods rely heavily on annotated data, and struggle with effectively disentangling timbre and style, leading to challenges in achieving controllable generation, especially in zero-shot scenarios. To address these issues, we propose Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. Vevo operates in two core stages: (1) Content-Style Modeling: Given either text or speech's content tokens as input, we utilize an autoregressive transformer to generate the content-style tokens, which is prompted by a style reference; (2) Acoustic Modeling: Given the content-style tokens as input, we employ a flow-matching transformer to produce acoustic representations, which is prompted by a timbre reference. To obtain the content and content-style tokens of speech, we design a fully self-supervised approach that progressively decouples the timbre, style, and linguistic content of speech. Specifically, we adopt VQ-VAE as the tokenizer for the continuous hidden features of HuBERT. We treat the vocabulary size of the VQ-VAE codebook as the information bottleneck, and adjust it carefully to obtain the disentangled speech representations. Solely self-supervised trained on 60K hours of audiobook speech data, without any fine-tuning on style-specific corpora, Vevo matches or surpasses existing methods in accent and emotion conversion tasks. Additionally, Vevo's effectiveness in zero-shot voice conversion and text-to-speech tasks further demonstrates its strong generalization and versatility. Audio samples are available at https://versavoice.github.io.