mlsanityData to 14 Oct 2026

Speech Synthesis with Mixed Emotions

arXiv 2208.05890 ↗First posted 11 Aug 2022cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Emotional speech synthesis aims to synthesize human voices with various emotional effects. The current studies are mostly focused on imitating an averaged style belonging to a specific emotion type. In this paper, we seek to generate speech with a mixture of emotions at run-time. We propose a novel formulation that measures the relative difference between the speech samples of different emotions. We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework. During the training, the framework does not only explicitly characterize emotion styles, but also explores the ordinal nature of emotions by quantifying the differences with other emotions. At run-time, we control the model to produce the desired emotion mixture by manually defining an emotion attribute vector. The objective and subjective evaluations have validated the effectiveness of the proposed framework. To our best knowledge, this research is the first study on modelling, synthesizing, and evaluating mixed emotions in speech.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 42 of its 136 references that are in the map. First posted on arXiv 11 Aug 2022.

Builds on

42 of 136 references.

References · 42

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  2. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  3. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  4. 2021A Survey on Neural Speech Synthesiscited by 184
  5. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  6. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 121
  7. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 115
  8. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 112
  9. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  10. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  11. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  12. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
30 more
  1. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  2. 2020Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody priorcited by 33
  3. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 32
  4. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  5. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  6. 2019Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretrainingcited by 29
  7. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  8. 2022MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesiscited by 29
  9. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  10. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  11. 2020Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Datacited by 26
  12. 2019CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Networkcited by 25
  13. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  14. 2020Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversioncited by 22
  15. 2019Exploring Transfer Learning for Low Resource Emotional TTScited by 21
  16. 2019End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Trainingcited by 19
  17. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  18. 2020VAW-GAN for Disentanglement and Recomposition of Emotional Elements in Speechcited by 18
  19. 2021Towards Multi-Scale Style Control for Expressive Speech Synthesiscited by 18
  20. 2020Transfer Learning from Speech Synthesis to Voice Conversion with Non-Parallel Training Datacited by 17
  21. 2021Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Trainingcited by 17
  22. 2019Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNetcited by 15
  23. 2018Neural Speech Synthesis with Transformer Networkcited by 14
  24. 2020NAUTILUS: a Versatile Voice Cloning Systemcited by 13
  25. 2021Model architectures to extrapolate emotional expressions in DNN-based text-to-speechcited by 6
  26. 2020A Siamese Neural Network with Modified Distance Loss For Transfer Learning in Speech Emotion Recognitioncited by 5
  27. 2020Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglementcited by 5
  28. 2019Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speechcited by 3
  29. 2020Recognition-Synthesis Based Non-Parallel Voice Conversion with Adversarial Learningcited by 2
  30. 2019Adjusting Pleasure-Arousal-Dominance for Continuous Emotional Text-to-speech Synthesizercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 10; 2025: 8; 2026: 4.

28 papers.

Sort:

By area:

  1. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  2. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 24
  3. 2023EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesiscited by 19
  4. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  5. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  6. 2023ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Modelscited by 14
  7. 2025Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Designcited by 11
  8. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 9
  9. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  10. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  11. 2024Affective Computing Has Changed: The Foundation Model Disruptioncited by 7
  12. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
16 more
  1. 2025Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotionscited by 4
  2. 2023In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesiscited by 3
  3. 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Modelscited by 3
  4. 2025ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speechcited by 3
  5. 2026A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Modelscited by 3
  6. 2022Semi-supervised learning for continuous emotional intensity controllable speech synthesis with disentangled representationscited by 2
  7. 2024Phonetic Enhanced Language Modeling for Text-to-Speech Synthesiscited by 2
  8. 2025Generative Adversarial Network based Voice Conversion: Techniques, Challenges, and Recent Advancementscited by 2
  9. 2025UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speechcited by 2
  10. 2025Fast-VGAN: Lightweight Voice Conversion with Explicit Control of F0 and Duration Parameterscited by 2
  11. 2024Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Modelcited by 1
  12. 2023CASEIN: Cascading Explicit and Implicit Control for Fine-grained Emotion Intensity Regulationcited by 0
  13. 2025HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speechcited by 0
  14. 2025EM2LDL: A Multilingual Speech Corpus for Mixed Emotion Recognition through Label Distribution Learningcited by 0
  15. 2026TargetSEC: Plug-and-Play In-the-Wild Speech Emotion Conversion via Arousal-Conditioned Latent Style Diffusioncited by 0
  16. 2026EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generationcited by 0