mlsanityData to 13 Oct 2026

PromptTTS: Controllable Text-to-Speech with Text Descriptions

arXiv 2211.12171 ↗First posted 22 Nov 2022eess.AS cs.CL cs.LGcited by 108

Not on the map. Why?

Abstract (arXiv)

Using a text description as prompt to guide the generation of text or images (e.g., GPT-3 or DALLE-2) has drawn wide attention recently. Beyond text and image generation, in this work, we explore the possibility of utilizing text descriptions to guide speech synthesis. Thus, we develop a text-to-speech (TTS) system (dubbed as PromptTTS) that takes a prompt with both style and content descriptions as input to synthesize the corresponding speech. Specifically, PromptTTS consists of a style encoder and a content encoder to extract the corresponding representations from the prompt, and a speech decoder to synthesize speech according to the extracted style and content representations. Compared with previous works in controllable TTS that require users to have acoustic knowledge to understand style factors such as prosody and pitch, PromptTTS is more user-friendly since text descriptions are a more natural way to express speech style (e.g., ''A lady whispers to her friend slowly''). Given that there is no TTS dataset with prompts, to benchmark the task of PromptTTS, we construct and release a dataset containing prompts with style and content information and the corresponding speech. Experiments show that PromptTTS can generate speech with precise style control and high speech quality. Audio samples and our dataset are publicly available.

Key facts. Cited by 108 papers in the map (data through 13 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

16 of 23 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  5. 2016Generative Adversarial Text to Image Synthesiscited by 735
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 715
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 643
  8. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  9. 2021A Survey on Neural Speech Synthesiscited by 182
  10. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  11. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  12. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
4 more
  1. 2019Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversioncited by 26
  2. 2021FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesiscited by 12
  3. 2020Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioningcited by 8
  4. 2022Unsupervised word-level prosody tagging for controllable speech synthesiscited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 17; 2024: 29; 2025: 33; 2026: 29.

108 papers.

Sort:

By area:

  1. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  2. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  3. 2026Qwen3-TTS Technical Reportcited by 114
  4. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  5. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  6. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  7. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 48
  8. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
  9. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 44
  10. 2025Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learningcited by 42
  11. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  12. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
38 more
  1. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 32
  2. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speechcited by 30
  3. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  4. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  5. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  6. 2024SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Descriptioncited by 20
  7. 2023PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Promptscited by 18
  8. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
  9. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
  10. 2024LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioningcited by 15
  11. 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systemscited by 15
  12. 2023EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speechcited by 14
  13. 2025SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuningcited by 14
  14. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
  15. 2024FLEURS-R: A Restored Multilingual Speech Corpus for Generation Taskscited by 13
  16. 2024Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Promptcited by 12
  17. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  18. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  19. 2024Voice Attribute Editing with Text Promptcited by 9
  20. 2024Controlling Emotion in Text-to-Speech with Natural Language Promptscited by 9
  21. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 8
  22. 2023StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Modelscited by 7
  23. 2024DLPO: Diffusion Model Loss-Guided Reinforcement Learning for Fine-Tuning Text-to-Speech Diffusion Modelscited by 7
  24. 2023SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizercited by 6
  25. 2023Towards General-Purpose Text-Instruction-Guided Voice Conversioncited by 6
  26. 2024SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioningcited by 6
  27. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  28. 2025InstructAudio: Unified speech and music generation with natural language instructioncited by 5
  29. 2026ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesiscited by 5
  30. 2026MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestrationcited by 5
  31. 2023ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddingscited by 4
  32. 2024UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Promptscited by 4
  33. 2024Exploring speech style spaces with language models: Emotional TTS without emotion labelscited by 4
  34. 2024Description-based Controllable Text-to-Speech with Cross-Lingual Voice Controlcited by 4
  35. 2025FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Stylescited by 4
  36. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Modelscited by 3
  37. 2024Expressivity and Speech Synthesiscited by 3
  38. 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Modelscited by 3

50 most cited of 108 All 108 (JSON)