mlsanityData to 14 Oct 2026

Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions

arXiv 2506.02742 ↗First posted 3 Jun 2025eess.AS cs.AI cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Existing expressive text-to-speech (TTS) systems primarily model a limited set of categorical emotions, whereas human conversations extend far beyond these predefined emotions, making it essential to explore more diverse emotional speech generation for more natural interactions. To bridge this gap, this paper proposes a novel prompt-unseen-emotion (PUE) approach to generate unseen emotional speech via emotion-guided prompt learning. PUE is trained utilizing an LLM-TTS architecture to ensure emotional consistency between categorical emotion-relevant prompts and emotional speech, allowing the model to quantitatively capture different emotion weightings per utterance. During inference, mixed emotional speech can be generated by flexibly adjusting emotion proportions and leveraging LLM contextual knowledge, enabling the model to quantify different emotional styles. Our proposed PUE successfully facilitates expressive speech synthesis of unseen emotions in a zero-shot setting.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

28 of 37 references.

References · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  4. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  5. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  6. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  7. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 115
  8. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
  9. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  10. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  11. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  12. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 32
16 more
  1. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  2. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  3. 2019Emotional speech synthesis with rich and granularized controlcited by 28
  4. 2022Speech Synthesis with Mixed Emotionscited by 28
  5. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  6. 2021Expressive Text-to-Speech using Style Tagcited by 20
  7. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  8. 2023EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesiscited by 19
  9. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  10. 2023Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modelingcited by 17
  11. 2023EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speechcited by 15
  12. 2024TTSlow: Slow Down Text-to-Speech with Efficiency Robustness Evaluationscited by 13
  13. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  14. 2024Hierarchical Control of Emotion Rendering in Speech Synthesiscited by 3
  15. 2022Mixed-EVC: Mixed Emotion Synthesis and Control in Voice Conversioncited by 2
  16. 2022Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencodercited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 9
  2. 2025MultiGen: Child-Friendly Multilingual Speech Generator with LLMscited by 1
  3. 2026Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Modelscited by 1
  4. 2026Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTScited by 0