# PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions

arXiv: https://arxiv.org/abs/2305.19522 · HTML: https://mlsanity.com/paper/2305.19522/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2023.

## Built on by (most cited first, 30 of 34)

- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://mlsanity.com/paper/2402.01912/) (2024)
- [TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models](https://mlsanity.com/paper/2308.14430/) (2023)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning](https://mlsanity.com/paper/2504.05108/) (2025)
- [PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions](https://mlsanity.com/paper/2309.08140/) (2023)
- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [VoiceLDM: Text-to-Speech with Environmental Context](https://mlsanity.com/paper/2309.13664/) (2023)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback](https://mlsanity.com/paper/2406.00654/) (2024)
- [Scaling Rich Style-Prompted Text-to-Speech Datasets](https://mlsanity.com/paper/2503.04713/) (2025)
- [PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts](https://mlsanity.com/paper/2309.09262/) (2023)
- [Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators](https://mlsanity.com/paper/2402.01708/) (2024)
- [ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control](https://mlsanity.com/paper/2406.01205/) (2024)
- [InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems](https://mlsanity.com/paper/2506.16381/) (2025)
- [LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning](https://mlsanity.com/paper/2406.07969/) (2024)
- [Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization](https://mlsanity.com/paper/2407.02243/) (2024)
- [VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models](https://mlsanity.com/paper/2504.02386/) (2025)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [Voice Attribute Editing with Text Prompt](https://mlsanity.com/paper/2404.08857/) (2024)
- [Controlling Emotion in Text-to-Speech with Natural Language Prompts](https://mlsanity.com/paper/2406.06406/) (2024)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)
- [Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction](https://mlsanity.com/paper/2312.10305/) (2023)
- [ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2601.03632/) (2026)
- [Description-based Controllable Text-to-Speech with Cross-Lingual Voice Control](https://mlsanity.com/paper/2409.17452/) (2024)
- [Gender Bias in Instruction-Guided Speech Synthesis Models](https://mlsanity.com/paper/2502.05649/) (2025)
- [Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Models](https://mlsanity.com/paper/2409.06451/) (2024)
- [VoXtream2: Full-stream TTS with dynamic speaking rate control](https://mlsanity.com/paper/2603.13518/) (2026)
- [AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing](https://mlsanity.com/paper/2609.08936/) (2026)

## Abstract (arXiv)

Style transfer TTS has shown impressive performance in recent years. However, style control is often restricted to systems built on expressive speech recordings with discrete style categories. In practical situations, users may be interested in transferring style by typing text descriptions of desired styles, without the reference speech in the target style. The text-guided content generation techniques have drawn wide attention recently. In this work, we explore the possibility of controllable style transfer with natural language descriptions. To this end, we propose PromptStyle, a text prompt-guided cross-speaker style transfer system. Specifically, PromptStyle consists of an improved VITS and a cross-modal style encoder. The cross-modal style encoder constructs a shared space of stylistic and semantic representation through a two-stage training process. Experiments show that PromptStyle can achieve proper style transfer with text prompts while maintaining relatively high stability and speaker similarity. Audio samples are available in our demo page.