mlsanityData to 14 Oct 2026

CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation

arXiv 2608.11590 ↗First posted 12 Aug 2026cs.SD cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves comparable performance to large-scale baselines with only 43.51 million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications. Demo page is available at https://haoweilou.github.io/CookVoice/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

23 of 29 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  7. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  8. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  9. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  10. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  11. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  12. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 121
11 more
  1. 2025Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglementcited by 35
  2. 2024GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Taskscited by 30
  3. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 28
  4. 2023StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesiscited by 21
  5. 2024TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Controlcited by 20
  6. 2022Common Phone: A Multilingual Dataset for Robust Acoustic Modellingcited by 7
  7. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  8. 2025Generalized Multilingual Text-to-Speech Generation with Language-Aware Style Adaptationcited by 3
  9. 2025ParaStyleTTS: Toward Efficient and Robust Paralinguistic Style Control for Expressive Text-to-Speech Generationcited by 3
  10. 2026ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speechcited by 1
  11. 2026AutoSIFT: Automatic Style Sifting for Controllable Speech Generation with Arbitrary Style Infillingcited by 1

Built on by

Not cited yet.