# Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis

arXiv: https://arxiv.org/abs/2011.08477 · HTML: https://mlsanity.com/paper/2011.08477/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2020.

## Built on by (most cited first, 29 of 29)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance](https://mlsanity.com/paper/2211.09496/) (2022)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [Towards Multi-Scale Style Control for Expressive Speech Synthesis](https://mlsanity.com/paper/2104.03521/) (2021)
- [EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model](https://mlsanity.com/paper/2106.09317/) (2021)
- [Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling](https://mlsanity.com/paper/2106.06233/) (2021)
- [Cross-Speaker Emotion Transfer for Low-Resource Text-to-Speech Using Non-Parallel Voice Conversion with Pitch-Shift Data Augmentation](https://mlsanity.com/paper/2204.10020/) (2022)
- [DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignment](https://mlsanity.com/paper/2401.08095/) (2024)
- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis](https://mlsanity.com/paper/2601.22873/) (2026)
- [Accurate Emotion Strength Assessment for Seen and Unseen Speech Based on Data-Driven Deep Learning](https://mlsanity.com/paper/2206.07229/) (2022)
- [Fine-grained Emotional Control of Text-To-Speech: Learning To Rank Inter- And Intra-Class Emotion Intensities](https://mlsanity.com/paper/2303.01508/) (2023)
- [Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions](https://mlsanity.com/paper/2506.02742/) (2025)
- [Emotion Selectable End-to-End Text-based Speech Editing](https://mlsanity.com/paper/2212.10191/) (2022)
- [Controllable Accented Text-to-Speech Synthesis](https://mlsanity.com/paper/2209.10804/) (2022)
- [Semi-supervised learning for continuous emotional intensity controllable speech synthesis with disentangled representations](https://mlsanity.com/paper/2211.06160/) (2022)
- [Cross-speaker Emotion Transfer by Manipulating Speech Style Latents](https://mlsanity.com/paper/2303.08329/) (2023)
- [Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Models](https://mlsanity.com/paper/2603.17231/) (2026)
- [StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesis](https://mlsanity.com/paper/2110.03156/) (2021)
- [Audio Deepfake Attribution: An Initial Dataset and Investigation](https://mlsanity.com/paper/2208.10489/) (2022)
- [A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysis](https://mlsanity.com/paper/2309.11849/) (2023)
- [MultiGen: Child-Friendly Multilingual Speech Generator with LLMs](https://mlsanity.com/paper/2508.08715/) (2025)
- [Multilingual Emotion Neurons in Large Audio-Language Models](https://mlsanity.com/paper/2608.08772/) (2026)
- [CASEIN: Cascading Explicit and Implicit Control for Fine-grained Emotion Intensity Regulation](https://mlsanity.com/paper/2307.00020/) (2023)

## Abstract (arXiv)

This paper proposes a unified model to conduct emotion transfer, control and prediction for sequence-to-sequence based fine-grained emotional speech synthesis. Conventional emotional speech synthesis often needs manual labels or reference audio to determine the emotional expressions of synthesized speech. Such coarse labels cannot control the details of speech emotion, often resulting in an averaged emotion expression delivery, and it is also hard to choose suitable reference audio during inference. To conduct fine-grained emotion expression generation, we introduce phoneme-level emotion strength representations through a learned ranking function to describe the local emotion details, and the sentence-level emotion category is adopted to render the global emotions of synthesized speech. With the global render and local descriptors of emotions, we can obtain fine-grained emotion expressions from reference audio via its emotion descriptors (for transfer) or directly from phoneme-level manual labels (for control). As for the emotional speech synthesis with arbitrary text inputs, the proposed model can also predict phoneme-level emotion expressions from texts, which does not require any reference audio or manual label.