# Emotional Prosody Control for Speech Generation

arXiv: https://arxiv.org/abs/2111.04730 · HTML: https://mlsanity.com/paper/2111.04730/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 7 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Deep Voice: Real-time Neural Text-to-Speech](https://mlsanity.com/paper/1702.07825/) (2017)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](https://mlsanity.com/paper/1807.07281/) (2018)
- [Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning](https://mlsanity.com/paper/1710.07654/) (2017)
- [Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1904.03446/) (2019)
- [CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech](https://mlsanity.com/paper/2004.14617/) (2020)
- [Semi-Supervised Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1910.01709/) (2019)
- [Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis](https://mlsanity.com/paper/1906.03402/) (2019)

## Built on by (most cited first, 12 of 12)

- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models](https://mlsanity.com/paper/2305.13831/) (2023)
- [Fine-grained Emotional Control of Text-To-Speech: Learning To Rank Inter- And Intra-Class Emotion Intensities](https://mlsanity.com/paper/2303.01508/) (2023)
- [ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representations](https://mlsanity.com/paper/2303.01261/) (2023)
- [Exploring speech style spaces with language models: Emotional TTS without emotion labels](https://mlsanity.com/paper/2405.11413/) (2024)
- [Prosody Analysis of Audiobooks](https://mlsanity.com/paper/2310.06930/) (2023)
- [Controllable Accented Text-to-Speech Synthesis](https://mlsanity.com/paper/2209.10804/) (2022)
- [UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2505.10599/) (2025)
- [Fast-VGAN: Lightweight Voice Conversion with Explicit Control of F0 and Duration Parameters](https://mlsanity.com/paper/2507.04817/) (2025)
- [PRESENT: Zero-Shot Text-to-Prosody Control](https://mlsanity.com/paper/2408.06827/) (2024)
- [Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis](https://mlsanity.com/paper/2609.03992/) (2026)

## Abstract (arXiv)

Machine-generated speech is characterized by its limited or unnatural emotional variation. Current text to speech systems generates speech with either a flat emotion, emotion selected from a predefined set, average variation learned from prosody sequences in training data or transferred from a source style. We propose a text to speech(TTS) system, where a user can choose the emotion of generated speech from a continuous and meaningful emotion space (Arousal-Valence space). The proposed TTS system can generate speech from the text in any speaker's style, with fine control of emotion. We show that the system works on emotion unseen during training and can scale to previously unseen speakers given his/her speech sample. Our work expands the horizon of the state-of-the-art FastSpeech2 backbone to a multi-speaker setting and gives it much-coveted continuous (and interpretable) affective control, without any observable degradation in the quality of the synthesized speech.