WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS
Not on the map. Why?
Abstract (arXiv)
While recent Large Language Model (LLM)-based Text-to-Speech (TTS) systems have achieved remarkable naturalness, they predominantly rely on implicit end-to-end generation paradigms, resulting in coarse-grained control. In scenarios demanding precise stylistic interventions and strict temporal alignment, such as audiobook narration and video dubbing, the inability to explicitly manipulate word-level acoustic attributes remains a critical bottleneck. This limitation is primarily amplified by the severe scarcity of fine-grained annotated datasets and the architectural challenge of integrating multi-dimensional control signals into discrete autoregressive generation. To address this, we propose a unified framework for highly precise word-level control. First, we construct WordVoice-5A, a massive 4.7k-hour bilingual dataset featuring five-dimensional word-level annotations (duration, boundary, energy, pitch and tone) developed through a rigorous linguistically-guided pipeline. Second, we introduce WordVoice to transform the implicit generation process into an explicit, highly controllable paradigm. Specifically, we introduce a bound-token mechanism within the LLM to formulate an explicit ``acoustic planning'' process, enabling adaptive multi-task prosodic planning and flexible manual intervention. Furthermore, we augment the token-to-waveform stage with a fine-grained acoustic modulation module, bridging the resolution gap to strictly align word-level attributes between highly compressed discrete tokens and continuous waveforms. Extensive experiments demonstrate that WordVoice achieves superior, decoupled control over multiple acoustic dimensions while maintaining competitive zero-shot synthesis stability. The code and audio samples are publicly available at https://xxh333.github.io/wordvoice-demo/.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 7 Jul 2026.
Builds on
27 of 36 references.
References · 27
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
- 2023Scaling Speech Technology to 1,000+ Languagescited by 411
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
- 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
- 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
- 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
- 2026Qwen3-TTS Technical Reportcited by 118
- 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 112
- 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
15 more
- 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
- 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
- 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
- 2024SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Descriptioncited by 20
- 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
- 2026MOSS-TTS Technical Reportcited by 14
- 2025Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesiscited by 7
- 2026LEMAS: Large A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Modelscited by 7
- 2025FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Stylescited by 4
- 2025Scalable Controllable Accented TTScited by 4
- 2025HD-PPT: Hierarchical Decoding of Content- and Prompt-Preference Tokens for Instruction-based TTScited by 4
- 2025Deep Dubbing: End-to-End Auto-Audiobook System with Text-to-Timbre and Context-Aware Instruct-TTScited by 3
- 2026MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Controlcited by 3
- 2026DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronizationcited by 2
- 2026HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speechcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.