# Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesis

arXiv: https://arxiv.org/abs/2405.09171 · HTML: https://mlsanity.com/paper/2405.09171/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2024.

## Built on by (most cited first, 7 of 7)

- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions](https://mlsanity.com/paper/2409.16681/) (2024)
- [EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis](https://mlsanity.com/paper/2601.22873/) (2026)
- [Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Models](https://mlsanity.com/paper/2409.06451/) (2024)
- [UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2505.10599/) (2025)
- [Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis](https://mlsanity.com/paper/2609.03992/) (2026)
- [Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters](https://mlsanity.com/paper/2610.05575/) (2026)

## Abstract (arXiv)

It remains a challenge to effectively control the emotion rendering in text-to-speech (TTS) synthesis. Prior studies have primarily focused on learning a global prosodic representation at the utterance level, which strongly correlates with linguistic prosody. Our goal is to construct a hierarchical emotion distribution (ED) that effectively encapsulates intensity variations of emotions at various levels of granularity, encompassing phonemes, words, and utterances. During TTS training, the hierarchical ED is extracted from the ground-truth audio and guides the predictor to establish a connection between emotional and linguistic prosody. At run-time inference, the TTS model generates emotional speech and, at the same time, provides quantitative control of emotion over the speech constituents. Both objective and subjective evaluations validate the effectiveness of the proposed framework in terms of emotion prediction and control.