# DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions

arXiv: https://arxiv.org/abs/2501.04256 · HTML: https://mlsanity.com/paper/2501.04256/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2025.

## Built on by (most cited first, 6 of 6)

- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models](https://mlsanity.com/paper/2507.20091/) (2025)
- [CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance](https://mlsanity.com/paper/2509.19883/) (2025)
- [Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis](https://mlsanity.com/paper/2512.18699/) (2025)
- [Improving French Synthetic Speech Quality via SSML Prosody Control](https://mlsanity.com/paper/2508.17494/) (2025)
- [Imitation Learning for Elder-Facing Speech Synthesis](https://mlsanity.com/paper/2606.21053/) (2026)

## Abstract (arXiv)

Controlling text-to-speech (TTS) systems to synthesize speech with the prosodic characteristics expected by users has attracted much attention. To achieve controllability, current studies focus on two main directions: (1) using reference speech as prosody prompt to guide speech synthesis, and (2) using natural language descriptions to control the generation process. However, finding reference speech that exactly contains the prosody that users want to synthesize takes a lot of effort. Description-based guidance in TTS systems can only determine the overall prosody, which has difficulty in achieving fine-grained prosody control over the synthesized speech. In this paper, we propose DrawSpeech, a sketch-conditioned diffusion model capable of generating speech based on any prosody sketches drawn by users. Specifically, the prosody sketches are fed to DrawSpeech to provide a rough indication of the expected prosody trends. DrawSpeech then recovers the detailed pitch and energy contours based on the coarse sketches and synthesizes the desired speech. Experimental results show that DrawSpeech can generate speech with a wide variety of prosody and can precisely control the fine-grained prosody in a user-friendly manner. Our implementation and audio samples are publicly available.