mlsanityData to 7 Oct 2026

EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis

arXiv 2306.00648 ↗First posted 1 Jun 2023cs.SD eess.AScited by 19

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

There has been significant progress in emotional Text-To-Speech (TTS) synthesis technology in recent years. However, existing methods primarily focus on the synthesis of a limited number of emotion types and have achieved unsatisfactory performance in intensity control. To address these limitations, we propose EmoMix, which can generate emotional speech with specified intensity or a mixture of emotions. Specifically, EmoMix is a controllable emotional TTS model based on a diffusion probabilistic model and a pre-trained speech emotion recognition (SER) model used to extract emotion embedding. Mixed emotion synthesis is achieved by combining the noises predicted by diffusion model conditioned on different emotions during only one sampling process at the run-time. We further apply the Neutral and specific primary emotion mixed in varying degrees to control intensity. Experimental results validate the effectiveness of EmoMix for synthesizing mixed emotion and intensity control.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 21 of its 42 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

21 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
  2. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 19

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  8. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  9. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  10. 2022MagicMix: Semantic Mixing with Diffusion Modelscited by 63
  11. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  12. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
7 more
  1. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  2. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  3. 2022MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesiscited by 29
  4. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  5. 2022Speech Synthesis with Mixed Emotionscited by 28
  6. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  7. 2022Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesiscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 9; 2026: 2.

19 papers.

Sort:

By area:

  1. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 33
  2. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  3. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  4. 2024ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesiscited by 9
  5. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  6. 2023EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Datacited by 7
  7. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
  8. 2025MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactionscited by 7
  9. 2025Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotionscited by 4
  10. 2025RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTScited by 4
  11. 2025UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speechcited by 2
  12. 2023AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesiscited by 1
7 more
  1. 2025Evaluating Human-LLM Representation Alignment: A Case Study on Affective Sentence Generation for Augmentative and Alternative Communicationcited by 1
  2. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesiscited by 1
  3. 2026AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesiscited by 1
  4. 2023Voice Conversion with Denoising Diffusion Probabilistic GAN Modelscited by 0
  5. 2024Daisy-TTS: Simulating Wider Spectrum of Emotions via Prosody Embedding Decompositioncited by 0
  6. 2025A Novel Data Augmentation Approach for Automatic Speaking Assessment on Opinion Expressionscited by 0
  7. 2026DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hierarchical Control of Emotion Rendering in Speech Synthesis
  2. 2025EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering
  3. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering
  4. 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Models
  5. 2025Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization
  6. 2025TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet
  7. 2026TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion ↗
  8. 2026A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models