# EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech

arXiv: https://arxiv.org/abs/2406.07803 · HTML: https://mlsanity.com/paper/2406.07803/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 26 of 26)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions](https://mlsanity.com/paper/2305.19522/) (2023)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [Semi-Supervised Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1910.01709/) (2019)
- [The ins and outs of speaker recognition: lessons from VoxSRC 2020](https://mlsanity.com/paper/2010.15809/) (2020)
- [Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis](https://mlsanity.com/paper/2109.06733/) (2021)
- [GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis](https://mlsanity.com/paper/2106.15153/) (2021)
- [SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech](https://mlsanity.com/paper/2204.11792/) (2022)
- [DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training](https://mlsanity.com/paper/2307.16549/) (2023)
- [ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models](https://mlsanity.com/paper/2305.13831/) (2023)
- [Emotional Prosody Control for Speech Generation](https://mlsanity.com/paper/2111.04730/) (2021)
- [Language Model-Based Emotion Prediction Methods for Emotional Speech Synthesis Systems](https://mlsanity.com/paper/2206.15067/) (2022)
- [HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer](https://mlsanity.com/paper/2307.16171/) (2023)
- [Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesis](https://mlsanity.com/paper/2207.01198/) (2022)
- [Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTS](https://mlsanity.com/paper/2207.06000/) (2022)
- [Explicit Intensity Control for Accented Text-to-speech](https://mlsanity.com/paper/2210.15364/) (2022)
- [Learning Emotional Representations from Imbalanced Speech Data for Speech Emotion Recognition and Emotional Text-to-Speech](https://mlsanity.com/paper/2306.05709/) (2023)

## Built on by (most cited first, 24 of 24)

- [Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2407.12229/) (2024)
- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering](https://mlsanity.com/paper/2602.03420/) (2026)
- [Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions](https://mlsanity.com/paper/2409.16681/) (2024)
- [EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis](https://mlsanity.com/paper/2601.22873/) (2026)
- [DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech](https://mlsanity.com/paper/2505.19687/) (2025)
- [Beyond Global Emotion: Fine-Grained Emotional Speech Synthesis with Dynamic Word-Level Modulation](https://mlsanity.com/paper/2509.20378/) (2025)
- [EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis](https://mlsanity.com/paper/2608.23791/) (2026)
- [EmoReg: Directional Latent Vector Modeling for Emotional Intensity Regularization in Diffusion-based Voice Conversion](https://mlsanity.com/paper/2412.20359/) (2024)
- [UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2505.10599/) (2025)
- [Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Models](https://mlsanity.com/paper/2603.17231/) (2026)
- [Spotlight-TTS: Spotlighting the Style via Voiced-Aware Style Extraction and Style Direction Adjustment for Expressive Text-to-Speech](https://mlsanity.com/paper/2505.20868/) (2025)
- [RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis](https://mlsanity.com/paper/2510.14628/) (2025)
- [HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech](https://mlsanity.com/paper/2606.28249/) (2026)
- [Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction](https://mlsanity.com/paper/2608.28970/) (2026)
- [EmoSphere-SER: Enhancing Speech Emotion Recognition Through Spherical Representation with Auxiliary Classification](https://mlsanity.com/paper/2505.19693/) (2025)
- [GamerAstra: Supporting 2D Non-Twitch Video Games for Blind and Low-Vision Players through a Multi-Agent Framework](https://mlsanity.com/paper/2506.22937/) (2025)
- [Emotion-Aware Speech Generation with Character-Specific Voices for Comics](https://mlsanity.com/paper/2509.15253/) (2025)
- [Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models](https://mlsanity.com/paper/2510.13293/) (2025)
- [Voiced-Aware Style Extraction and Style Direction Adjustment for Expressive Text-to-Speech](https://mlsanity.com/paper/2511.14824/) (2025)
- [DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech](https://mlsanity.com/paper/2606.00066/) (2026)
- [Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis](https://mlsanity.com/paper/2609.03992/) (2026)
- [Steering Follows Geometry, Not Labels: Emotion Directions in a Full-Duplex Speech Model](https://mlsanity.com/paper/2610.08887/) (2026)

## Abstract (arXiv)

Despite rapid advances in the field of emotional text-to-speech (TTS), recent studies primarily focus on mimicking the average style of a particular emotion. As a result, the ability to manipulate speech emotion remains constrained to several predefined labels, compromising the ability to reflect the nuanced variations of emotion. In this paper, we propose EmoSphere-TTS, which synthesizes expressive emotional speech by using a spherical emotion vector to control the emotional style and intensity of the synthetic speech. Without any human annotation, we use the arousal, valence, and dominance pseudo-labels to model the complex nature of emotion via a Cartesian-spherical transformation. Furthermore, we propose a dual conditional adversarial network to improve the quality of generated speech by reflecting the multi-aspect characteristics. The experimental results demonstrate the model ability to control emotional style and intensity with high-quality expressive speech.