Expressive TTS Driven by Natural Language Prompts Using Few Human Annotations
Not on the map. Why?
Abstract (arXiv)
Expressive text-to-speech (TTS) aims to synthesize speeches with human-like tones, moods, or even artistic attributes. Recent advancements in expressive TTS empower users with the ability to directly control synthesis style through natural language prompts. However, these methods often require excessive training with a significant amount of style-annotated data, which can be challenging to acquire. Moreover, they may have limited adaptability due to fixed style annotations. In this work, we present FreeStyleTTS (FS-TTS), a controllable expressive TTS model with minimal human annotations. Our approach utilizes a large language model (LLM) to transform expressive TTS into a style retrieval task. The LLM selects the best-matching style references from annotated utterances based on external style prompts, which can be raw input text or natural language style descriptions. The selected reference guides the TTS pipeline to synthesize speeches with the intended style. This innovative approach provides flexible, versatile, and precise style control with minimal human workload. Experiments on a Mandarin storytelling corpus demonstrate FS-TTS's proficiency in leveraging LLM's semantic inference ability to retrieve desired styles from either input text or user-defined descriptions. This results in synthetic speeches that are closely aligned with the specified styles.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 24 references that are in the map. First posted on arXiv 2 Nov 2023.
Builds on
22 of 24 references.
References · 22
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
- 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
- 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
- 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
- 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
10 more
- 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
- 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
- 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
- 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
- 2019Emotional speech synthesis with rich and granularized controlcited by 28
- 2019Exploring Transfer Learning for Low Resource Emotional TTScited by 21
- 2021Expressive Text-to-Speech using Style Tagcited by 20
- 2021Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesiscited by 10
- 2019Visualization and Interpretation of Latent Spaces for Controlling Expressive Speech Synthesis through Audio Analysiscited by 8
- 2022Unsupervised word-level prosody tagging for controllable speech synthesiscited by 7
Built on by
Citing papers per year: 2025: 1.
1 paper.