mlsanityData to 14 Oct 2026

WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

arXiv 2607.06461 ↗First posted 7 Jul 2026eess.AS cs.CL cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

While recent Large Language Model (LLM)-based Text-to-Speech (TTS) systems have achieved remarkable naturalness, they predominantly rely on implicit end-to-end generation paradigms, resulting in coarse-grained control. In scenarios demanding precise stylistic interventions and strict temporal alignment, such as audiobook narration and video dubbing, the inability to explicitly manipulate word-level acoustic attributes remains a critical bottleneck. This limitation is primarily amplified by the severe scarcity of fine-grained annotated datasets and the architectural challenge of integrating multi-dimensional control signals into discrete autoregressive generation. To address this, we propose a unified framework for highly precise word-level control. First, we construct WordVoice-5A, a massive 4.7k-hour bilingual dataset featuring five-dimensional word-level annotations (duration, boundary, energy, pitch and tone) developed through a rigorous linguistically-guided pipeline. Second, we introduce WordVoice to transform the implicit generation process into an explicit, highly controllable paradigm. Specifically, we introduce a bound-token mechanism within the LLM to formulate an explicit ``acoustic planning'' process, enabling adaptive multi-task prosodic planning and flexible manual intervention. Furthermore, we augment the token-to-waveform stage with a fine-grained acoustic modulation module, bridging the resolution gap to strictly align word-level attributes between highly compressed discrete tokens and continuous waveforms. Extensive experiments demonstrate that WordVoice achieves superior, decoupled control over multiple acoustic dimensions while maintaining competitive zero-shot synthesis stability. The code and audio samples are publicly available at https://xxh333.github.io/wordvoice-demo/.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

27 of 36 references.

References · 27

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  3. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  4. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  5. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  6. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  7. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  8. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  9. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  10. 2026Qwen3-TTS Technical Reportcited by 118
  11. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 112
  12. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
15 more
  1. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  2. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  3. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  4. 2024SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Descriptioncited by 20
  5. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  6. 2026MOSS-TTS Technical Reportcited by 14
  7. 2025Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesiscited by 7
  8. 2026LEMAS: Large A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Modelscited by 7
  9. 2025FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Stylescited by 4
  10. 2025Scalable Controllable Accented TTScited by 4
  11. 2025HD-PPT: Hierarchical Decoding of Content- and Prompt-Preference Tokens for Instruction-based TTScited by 4
  12. 2025Deep Dubbing: End-to-End Auto-Audiobook System with Text-to-Timbre and Context-Aware Instruct-TTScited by 3
  13. 2026MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Controlcited by 3
  14. 2026DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronizationcited by 2
  15. 2026HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speechcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.