mlsanityData to 19 Oct 2026

CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis

arXiv 2608.08362 ↗First posted 8 Aug 2026eess.AS cs.HC cs.SDcited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Recent Text-To-Speech (TTS) systems have achieved strong naturalness and zero-shot voice cloning performance, but fine-grained control of expressive speech at the word or phoneme level remains challenging. We propose CtrlSpeech, a controllable, expressive TTS framework with coarse-to-fine control. Built on the DiTAR architecture, CtrlSpeech combines global speaker conditioning with phone-aligned pitch, loudness, and duration signals, enabling localized prosodic control while preserving the target speaker's timbre. This design allows users to adjust expressive attributes at a fine temporal granularity, making speech refinement more flexible and controllable. Experimental results show that CtrlSpeech achieves competitive zero-shot TTS performance and improves controllability over expressive attributes, demonstrating its effectiveness for flexible and practical expressive speech synthesis.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 30 of its 36 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

30 of 36 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing15 Nov
  2. 2025Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs14 Oct
  3. 2025Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis26 Sep
  4. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis18 Sep
  5. 2025StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling14 Jun
  6. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training23 May
  7. 2025Scaling Rich Style-Prompted Text-to-Speech Datasets6 Mar
  8. 2025DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions8 Jan
  9. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  10. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  11. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer1 Sep
  12. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens7 Jul
7 more
  1. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Models4 Jun
  2. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  3. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotations2 Feb
  4. 2023PromptTTS 2: Describing and Generating Voices with Text Prompt5 Sep
  5. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt31 Jan
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  7. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptions22 Nov

Other references · 11

  1. 2022Classifier-Free Diffusion Guidancecited by 5,991
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  4. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
  5. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 497
  6. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 315
  7. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 268
  8. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 77
  9. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 58
  10. 2024VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modellingcited by 43
  11. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet
  2. 2026Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
  3. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey
  4. 2026FC-TTS: Style and Timbre Control in Zero-Shot Text-to-Speech with Disentangled Speech Representations
  5. 2026FineCombo-TTS: Collaborative and Precise Controllable Speech Synthesis Using Text Descriptions and Reference Speech
  6. 2026MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Control
  7. 2024EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Control
  8. 2026SCIC: Scope- and Codebook-Aware Instruction Conditioning for Speaker-Adapted Expressive TTS