# Exploring speech style spaces with language models: Emotional TTS without emotion labels

arXiv: https://arxiv.org/abs/2405.11413 · HTML: https://mlsanity.com/paper/2405.11413/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 18 May 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis](https://mlsanity.com/paper/1808.01410/) (2018)
- [Emotional End-to-End Neural Speech Synthesizer](https://mlsanity.com/paper/1711.05447/) (2017)
- [Expressive TTS Training with Frame and Style Reconstruction Loss](https://mlsanity.com/paper/2008.01490/) (2020)
- [Emotional speech synthesis with rich and granularized control](https://mlsanity.com/paper/1911.01635/) (2019)
- [Expressive Text-to-Speech using Style Tag](https://mlsanity.com/paper/2104.00436/) (2021)
- [End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Training](https://mlsanity.com/paper/1906.10859/) (2019)
- [Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition](https://mlsanity.com/paper/2010.13350/) (2020)
- [Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability](https://mlsanity.com/paper/2104.01408/) (2021)
- [EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model](https://mlsanity.com/paper/2106.09317/) (2021)
- [Emotional Prosody Control for Speech Generation](https://mlsanity.com/paper/2111.04730/) (2021)
- [Speech BERT Embedding For Improving Prosody in Neural TTS](https://mlsanity.com/paper/2106.04312/) (2021)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [Language Model-Based Emotion Prediction Methods for Emotional Speech Synthesis Systems](https://mlsanity.com/paper/2206.15067/) (2022)
- [Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTS](https://mlsanity.com/paper/2207.06000/) (2022)
- [MSStyleTTS: Multi-Scale Style Modeling with Hierarchical Context Information for Expressive Speech Synthesis](https://mlsanity.com/paper/2307.16012/) (2023)

## Built on by (most cited first, 4 of 4)

- [CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering](https://mlsanity.com/paper/2602.03420/) (2026)
- [Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Models](https://mlsanity.com/paper/2409.06451/) (2024)
- [NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion](https://mlsanity.com/paper/2511.00256/) (2025)
- [Making Social Platforms Accessible: Emotion-Aware Speech Generation with Integrated Text Analysis](https://mlsanity.com/paper/2410.19199/) (2024)

## Abstract (arXiv)

Many frameworks for emotional text-to-speech (E-TTS) rely on human-annotated emotion labels that are often inaccurate and difficult to obtain. Learning emotional prosody implicitly presents a tough challenge due to the subjective nature of emotions. In this study, we propose a novel approach that leverages text awareness to acquire emotional styles without the need for explicit emotion labels or text prompts. We present TEMOTTS, a two-stage framework for E-TTS that is trained without emotion labels and is capable of inference without auxiliary inputs. Our proposed method performs knowledge transfer between the linguistic space learned by BERT and the emotional style space constructed by global style tokens. Our experimental results demonstrate the effectiveness of our proposed framework, showcasing improvements in emotional accuracy and naturalness. This is one of the first studies to leverage the emotional correlation between spoken content and expressive delivery for emotional TTS.