mlsanityData to 14 Oct 2026

Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation

arXiv 2508.16332 ↗First posted 22 Aug 2025cs.SD cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Controllable human voice generation, particularly for expressive domains like singing, remains a significant challenge. This paper introduces Vevo2, a unified framework for controllable speech and singing voice generation. To tackle issues like the scarcity of annotated singing data and to enable flexible controllability, Vevo2 introduces two audio tokenizers: (1) a unified music-notation-free prosody tokenizer that captures prosody and melody from speech, singing, and even instrumental sounds, and (2) a unified content-style tokenizer that encodes linguistic content, prosody, and style for both speech and singing, while enabling timbre disentanglement. Vevo2 consists of an auto-regressive (AR) content-style modeling stage, which aims to enable controllability over text, prosody, and style, as well as a flow-matching acoustic modeling stage that allows for timbre control. Particularly, during the speech-singing joint training of the AR model, we propose both explicit and implicit prosody learning strategies to bridge speech and singing voice. Moreover, to further enhance the Vevo2's ability to follow text and prosody, we design a multi-objective post-training task that integrates both intelligibility and prosody similarity alignment. Experimental results show that the unified modeling in Vevo2 brings mutual benefits to both speech and singing voice generation. Additionally, Vevo2's effectiveness across a wide range of synthesis, conversion, and editing tasks for both speech and singing further demonstrates its strong generalization ability and versatility. Audio samples are are available at https://versasinger.github.io/.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 77 of its 107 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

77 of 107 references.

References · 77

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  12. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
38 more
  1. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2023Simple and Controllable Music Generationcited by 488
  4. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  6. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  7. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  9. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  10. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  11. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  12. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  13. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  14. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  15. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  16. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  17. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  18. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  19. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  20. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  21. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  22. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 115
  23. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  24. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  25. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  26. 2025YuE: Scaling Open Foundation Models for Long-Form Music Generationcited by 71
  27. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  28. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  29. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 47
  30. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 47
  31. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 43
  32. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  33. 2025Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglementcited by 35
  34. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  35. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 32
  36. 2024GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Taskscited by 30
  37. 2023Amphion: An Open-Source Audio, Music and Speech Generation Toolkitcited by 27
  38. 2023The Singing Voice Conversion Challenge 2023cited by 25

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  2. 2025YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidancecited by 5
  3. 2026Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidancecited by 1
  4. 2026CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generationcited by 0
  5. 2026Iterative Self-Learning for Expressive Text-to-Speech Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UniVoice: A Unified Model for Speech and Singing Voice Generation
  2. 2026S2Voice: Style-Aware Autoregressive Modeling with Enhanced Conditioning for Singing Style Conversion
  3. 2026VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models
  4. 2025Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation
  5. 2026Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation
  6. 2026Adapting Diffusion-Based Music Synthesis to Speech and Singing Voice Conversion
  7. 2026LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training
  8. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers