# EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance

arXiv: https://arxiv.org/abs/2211.09496 · HTML: https://mlsanity.com/paper/2211.09496/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)
- [Diff-TTS: A Denoising Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2104.01409/) (2021)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis](https://mlsanity.com/paper/2204.09934/) (2022)
- [Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset](https://mlsanity.com/paper/2010.14794/) (2020)
- [Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance](https://mlsanity.com/paper/2111.11755/) (2021)
- [BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis](https://mlsanity.com/paper/2203.13508/) (2022)
- [Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis](https://mlsanity.com/paper/2002.03785/) (2020)
- [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022)
- [VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature](https://mlsanity.com/paper/2204.00768/) (2022)
- [Emotional End-to-End Neural Speech Synthesizer](https://mlsanity.com/paper/1711.05447/) (2017)
- [Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis](https://mlsanity.com/paper/2011.08477/) (2020)
- [MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis](https://mlsanity.com/paper/2201.06460/) (2022)
- [Emotional speech synthesis with rich and granularized control](https://mlsanity.com/paper/1911.01635/) (2019)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [Expressive Text-to-Speech using Style Tag](https://mlsanity.com/paper/2104.00436/) (2021)
- [Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesis](https://mlsanity.com/paper/2105.13086/) (2021)
- [Unsupervised word-level prosody tagging for controllable speech synthesis](https://mlsanity.com/paper/2202.07200/) (2022)

## Built on by (most cited first, 30 of 37)

- [Matcha-TTS: A fast TTS architecture with conditional flow matching](https://mlsanity.com/paper/2309.03199/) (2023)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching](https://mlsanity.com/paper/2309.05027/) (2023)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis](https://mlsanity.com/paper/2306.00648/) (2023)
- [Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2407.12229/) (2024)
- [EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech](https://mlsanity.com/paper/2307.00024/) (2023)
- [ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models](https://mlsanity.com/paper/2305.13831/) (2023)
- [Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition](https://mlsanity.com/paper/2309.10294/) (2023)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesis](https://mlsanity.com/paper/2401.08166/) (2024)
- [EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Data](https://mlsanity.com/paper/2309.07828/) (2023)
- [StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models](https://mlsanity.com/paper/2311.16509/) (2023)
- [Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings](https://mlsanity.com/paper/2407.14056/) (2024)
- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis](https://mlsanity.com/paper/2601.22873/) (2026)
- [DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech](https://mlsanity.com/paper/2306.14145/) (2023)
- [UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts](https://mlsanity.com/paper/2404.18398/) (2024)
- [Can Emotion Fool Anti-spoofing?](https://mlsanity.com/paper/2505.23962/) (2025)
- [Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions](https://mlsanity.com/paper/2506.02742/) (2025)
- [RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS](https://mlsanity.com/paper/2512.04552/) (2025)
- [On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models](https://mlsanity.com/paper/2402.12423/) (2024)
- [Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis](https://mlsanity.com/paper/2512.18699/) (2025)
- [AUTODIFF: Autoregressive Diffusion Modeling for Structure-based Drug Design](https://mlsanity.com/paper/2404.02003/) (2024)
- [Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speech](https://mlsanity.com/paper/2409.16203/) (2024)
- [UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2505.10599/) (2025)
- [Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization](https://mlsanity.com/paper/2509.25416/) (2025)
- [AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis](https://mlsanity.com/paper/2308.08577/) (2023)
- [Expressive TTS Driven by Natural Language Prompts Using Few Human Annotations](https://mlsanity.com/paper/2311.01260/) (2023)
- [Humane Speech Synthesis through Zero-Shot Emotion and Disfluency Generation](https://mlsanity.com/paper/2404.01339/) (2024)

## Abstract (arXiv)

Although current neural text-to-speech (TTS) models are able to generate high-quality speech, intensity controllable emotional TTS is still a challenging task. Most existing methods need external optimizations for intensity calculation, leading to suboptimal results or degraded quality. In this paper, we propose EmoDiff, a diffusion-based TTS model where emotion intensity can be manipulated by a proposed soft-label guidance technique derived from classifier guidance. Specifically, instead of being guided with a one-hot vector for the specified emotion, EmoDiff is guided with a soft label where the value of the specified emotion and textitNeutral is set to α and 1-α respectively. The α here represents the emotion intensity and can be chosen from 0 to 1. Our experiments show that EmoDiff can precisely control the emotion intensity while maintaining high voice quality. Moreover, diverse speech with specified emotion intensity can be generated by sampling in the reverse denoising process.