# EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis arXiv: https://arxiv.org/abs/2306.00648 ยท HTML: https://mlsanity.com/paper/2306.00648/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Placed in Audio, speech and music, one of 14 directions in Diffusion models. - Builds on 21 of its 42 references that are in the map. - First posted on arXiv 1 Jun 2023. ## Placed in - Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/) ## Builds on (most cited first, 21 of 21) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016) - [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020) - [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021) - [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021) - [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018) - [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022) - [Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset](https://mlsanity.com/paper/2010.14794/) (2020) - [MagicMix: Semantic Mixing with Diffusion Models](https://mlsanity.com/paper/2210.16056/) (2022) - [EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance](https://mlsanity.com/paper/2211.09496/) (2022) - [Emotion Intensity and its Control for Emotional Voice Conversion](https://mlsanity.com/paper/2201.03967/) (2022) - [Controllable Emotion Transfer For End-to-End Speech Synthesis](https://mlsanity.com/paper/2011.08679/) (2020) - [Expressive TTS Training with Frame and Style Reconstruction Loss](https://mlsanity.com/paper/2008.01490/) (2020) - [MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis](https://mlsanity.com/paper/2201.06460/) (2022) - [Emotional speech synthesis with rich and granularized control](https://mlsanity.com/paper/1911.01635/) (2019) - [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022) - [Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition](https://mlsanity.com/paper/2010.13350/) (2020) - [Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesis](https://mlsanity.com/paper/2207.01198/) (2022) ## Built on by (most cited first, 19 of 19) - [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025) - [Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2407.12229/) (2024) - [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025) - [ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesis](https://mlsanity.com/paper/2401.08166/) (2024) - [Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions](https://mlsanity.com/paper/2409.16681/) (2024) - [EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Data](https://mlsanity.com/paper/2309.07828/) (2023) - [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024) - [MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions](https://mlsanity.com/paper/2507.10859/) (2025) - [Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions](https://mlsanity.com/paper/2506.02742/) (2025) - [RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS](https://mlsanity.com/paper/2512.04552/) (2025) - [UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2505.10599/) (2025) - [AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis](https://mlsanity.com/paper/2308.08577/) (2023) - [Evaluating Human-LLM Representation Alignment: A Case Study on Affective Sentence Generation for Augmentative and Alternative Communication](https://mlsanity.com/paper/2503.11881/) (2025) - [RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis](https://mlsanity.com/paper/2510.14628/) (2025) - [AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis](https://mlsanity.com/paper/2607.15755/) (2026) - [Voice Conversion with Denoising Diffusion Probabilistic GAN Models](https://mlsanity.com/paper/2308.14319/) (2023) - [Daisy-TTS: Simulating Wider Spectrum of Emotions via Prosody Embedding Decomposition](https://mlsanity.com/paper/2402.14523/) (2024) - [A Novel Data Augmentation Approach for Automatic Speaking Assessment on Opinion Expressions](https://mlsanity.com/paper/2506.04077/) (2025) - [DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech](https://mlsanity.com/paper/2606.00066/) (2026) ## Abstract (arXiv) There has been significant progress in emotional Text-To-Speech (TTS) synthesis technology in recent years. However, existing methods primarily focus on the synthesis of a limited number of emotion types and have achieved unsatisfactory performance in intensity control. To address these limitations, we propose EmoMix, which can generate emotional speech with specified intensity or a mixture of emotions. Specifically, EmoMix is a controllable emotional TTS model based on a diffusion probabilistic model and a pre-trained speech emotion recognition (SER) model used to extract emotion embedding. Mixed emotion synthesis is achieved by combining the noises predicted by diffusion model conditioned on different emotions during only one sampling process at the run-time. We further apply the Neutral and specific primary emotion mixed in varying degrees to control intensity. Experimental results validate the effectiveness of EmoMix for synthesizing mixed emotion and intensity control.