# PromptTTS: Controllable Text-to-Speech with Text Descriptions

arXiv: https://arxiv.org/abs/2211.12171 · HTML: https://mlsanity.com/paper/2211.12171/

## Key facts

- Cited by 110 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://mlsanity.com/paper/2104.12369/) (2021)
- [CPM-2: Large-scale Cost-effective Pre-trained Language Models](https://mlsanity.com/paper/2106.10715/) (2021)
- [Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1904.03446/) (2019)
- [FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesis](https://mlsanity.com/paper/2106.15123/) (2021)
- [Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioning](https://mlsanity.com/paper/2007.15281/) (2020)
- [Unsupervised word-level prosody tagging for controllable speech synthesis](https://mlsanity.com/paper/2202.07200/) (2022)

## Built on by (most cited first, 30 of 110)

- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://mlsanity.com/paper/2402.01912/) (2024)
- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models](https://mlsanity.com/paper/2308.14430/) (2023)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning](https://mlsanity.com/paper/2504.05108/) (2025)
- [DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data](https://mlsanity.com/paper/2409.20007/) (2024)
- [HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis](https://mlsanity.com/paper/2311.12454/) (2023)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment](https://mlsanity.com/paper/2406.18871/) (2024)
- [E3 TTS: Easy End-to-End Diffusion-based Text to Speech](https://mlsanity.com/paper/2311.00945/) (2023)
- [PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions](https://mlsanity.com/paper/2309.08140/) (2023)
- [VoiceLDM: Text-to-Speech with Environmental Context](https://mlsanity.com/paper/2309.13664/) (2023)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description](https://mlsanity.com/paper/2408.13608/) (2024)
- [PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts](https://mlsanity.com/paper/2309.09262/) (2023)
- [ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control](https://mlsanity.com/paper/2406.01205/) (2024)
- [Audio-Aware Large Language Models as Judges for Speaking Styles](https://mlsanity.com/paper/2506.05984/) (2025)
- [InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems](https://mlsanity.com/paper/2506.16381/) (2025)
- [EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech](https://mlsanity.com/paper/2307.00024/) (2023)
- [LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning](https://mlsanity.com/paper/2406.07969/) (2024)
- [SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning](https://mlsanity.com/paper/2504.09081/) (2025)
- [Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction](https://mlsanity.com/paper/2406.12707/) (2024)
- [FLEURS-R: A Restored Multilingual Speech Corpus for Generation Tasks](https://mlsanity.com/paper/2408.06227/) (2024)
- [Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt](https://mlsanity.com/paper/2403.11780/) (2024)
- [Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper](https://mlsanity.com/paper/2406.05806/) (2024)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)

## Abstract (arXiv)

Using a text description as prompt to guide the generation of text or images (e.g., GPT-3 or DALLE-2) has drawn wide attention recently. Beyond text and image generation, in this work, we explore the possibility of utilizing text descriptions to guide speech synthesis. Thus, we develop a text-to-speech (TTS) system (dubbed as PromptTTS) that takes a prompt with both style and content descriptions as input to synthesize the corresponding speech. Specifically, PromptTTS consists of a style encoder and a content encoder to extract the corresponding representations from the prompt, and a speech decoder to synthesize speech according to the extracted style and content representations. Compared with previous works in controllable TTS that require users to have acoustic knowledge to understand style factors such as prosody and pitch, PromptTTS is more user-friendly since text descriptions are a more natural way to express speech style (e.g., ''A lady whispers to her friend slowly''). Given that there is no TTS dataset with prompts, to benchmark the task of PromptTTS, we construct and release a dataset containing prompts with style and content information and the corresponding speech. Experiments show that PromptTTS can generate speech with precise style control and high speech quality. Audio samples and our dataset are publicly available.