# Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis

arXiv: https://arxiv.org/abs/1808.01410 · HTML: https://mlsanity.com/paper/1808.01410/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2018.

## Built on by (most cited first, 30 of 32)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [V2C: Visual Voice Cloning](https://mlsanity.com/paper/2111.12890/) (2021)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech](https://mlsanity.com/paper/2202.07816/) (2022)
- [DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021](https://mlsanity.com/paper/2110.12612/) (2021)
- [STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech](https://mlsanity.com/paper/2103.09474/) (2021)
- [EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model](https://mlsanity.com/paper/2106.09317/) (2021)
- [DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training](https://mlsanity.com/paper/2307.16549/) (2023)
- [Speaker Generation](https://mlsanity.com/paper/2111.05095/) (2021)
- [Expressive Neural Voice Cloning](https://mlsanity.com/paper/2102.00151/) (2021)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model](https://mlsanity.com/paper/2207.01718/) (2022)
- [Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions](https://mlsanity.com/paper/2409.16681/) (2024)
- [Word-Level Style Control for Expressive, Non-attentive Speech Synthesis](https://mlsanity.com/paper/2111.10173/) (2021)
- [Distribution augmentation for low-resource expressive text-to-speech](https://mlsanity.com/paper/2202.06409/) (2022)
- [ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddings](https://mlsanity.com/paper/2305.13724/) (2023)
- [StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis](https://mlsanity.com/paper/2312.12181/) (2023)
- [Exploring speech style spaces with language models: Emotional TTS without emotion labels](https://mlsanity.com/paper/2405.11413/) (2024)
- [On Prosody Modeling for ASR+TTS based Voice Conversion](https://mlsanity.com/paper/2107.09477/) (2021)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis](https://mlsanity.com/paper/2203.12201/) (2022)
- [Read the Room: Adapting a Robot's Voice to Ambient and Social Contexts](https://mlsanity.com/paper/2205.04952/) (2022)
- [Using multiple reference audios and style embedding constraints for speech synthesis](https://mlsanity.com/paper/2110.04451/) (2021)
- [Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech](https://mlsanity.com/paper/2205.04120/) (2022)
- [Time out of Mind: Generating Rate of Speech conditioned on emotion and speaker](https://mlsanity.com/paper/2301.12331/) (2023)
- [A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysis](https://mlsanity.com/paper/2309.11849/) (2023)
- [Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis](https://mlsanity.com/paper/2211.01327/) (2022)
- [Leveraging the Interplay Between Syntactic and Acoustic Cues for Optimizing Korean TTS Pause Formation](https://mlsanity.com/paper/2404.02592/) (2024)

## Abstract (arXiv)

Global Style Tokens (GSTs) are a recently-proposed method to learn latent disentangled representations of high-dimensional data. GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style. In this work, we introduce the Text-Predicted Global Style Token (TP-GST) architecture, which treats GST combination weights or style embeddings as "virtual" speaking style labels within Tacotron. TP-GST learns to predict stylistic renderings from text alone, requiring neither explicit labels during training nor auxiliary inputs for inference. We show that, when trained on a dataset of expressive speech, our system generates audio with more pitch and energy variation than two state-of-the-art baseline models. We further demonstrate that TP-GSTs can synthesize speech with background noise removed, and corroborate these analyses with positive results on human-rated listener preference audiobook tasks. Finally, we demonstrate that multi-speaker TP-GST models successfully factorize speaker identity and speaking style. We provide a website with audio samples for each of our findings.