# Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions

arXiv: https://arxiv.org/abs/2506.02742 · HTML: https://mlsanity.com/paper/2506.02742/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis](https://mlsanity.com/paper/2502.04128/) (2025)
- [EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance](https://mlsanity.com/paper/2211.09496/) (2022)
- [Controllable Emotion Transfer For End-to-End Speech Synthesis](https://mlsanity.com/paper/2011.08679/) (2020)
- [Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization](https://mlsanity.com/paper/2409.10157/) (2024)
- [Emotional End-to-End Neural Speech Synthesizer](https://mlsanity.com/paper/1711.05447/) (2017)
- [Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis](https://mlsanity.com/paper/2011.08477/) (2020)
- [Emotional speech synthesis with rich and granularized control](https://mlsanity.com/paper/1911.01635/) (2019)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech](https://mlsanity.com/paper/2302.04215/) (2023)
- [Expressive Text-to-Speech using Style Tag](https://mlsanity.com/paper/2104.00436/) (2021)
- [Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition](https://mlsanity.com/paper/2010.13350/) (2020)
- [EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis](https://mlsanity.com/paper/2306.00648/) (2023)
- [Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2407.12229/) (2024)
- [Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling](https://mlsanity.com/paper/2312.11947/) (2023)
- [EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech](https://mlsanity.com/paper/2307.00024/) (2023)
- [TTSlow: Slow Down Text-to-Speech with Efficiency Robustness Evaluations](https://mlsanity.com/paper/2407.01927/) (2024)
- [Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions](https://mlsanity.com/paper/2409.16681/) (2024)
- [Hierarchical Control of Emotion Rendering in Speech Synthesis](https://mlsanity.com/paper/2412.12498/) (2024)
- [Mixed-EVC: Mixed Emotion Synthesis and Control in Voice Conversion](https://mlsanity.com/paper/2210.13756/) (2022)
- [Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder](https://mlsanity.com/paper/2212.08329/) (2022)

## Built on by (most cited first, 4 of 4)

- [CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering](https://mlsanity.com/paper/2602.03420/) (2026)
- [MultiGen: Child-Friendly Multilingual Speech Generator with LLMs](https://mlsanity.com/paper/2508.08715/) (2025)
- [Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models](https://mlsanity.com/paper/2605.27376/) (2026)
- [Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS](https://mlsanity.com/paper/2606.09837/) (2026)

## Abstract (arXiv)

Existing expressive text-to-speech (TTS) systems primarily model a limited set of categorical emotions, whereas human conversations extend far beyond these predefined emotions, making it essential to explore more diverse emotional speech generation for more natural interactions. To bridge this gap, this paper proposes a novel prompt-unseen-emotion (PUE) approach to generate unseen emotional speech via emotion-guided prompt learning. PUE is trained utilizing an LLM-TTS architecture to ensure emotional consistency between categorical emotion-relevant prompts and emotional speech, allowing the model to quantitatively capture different emotion weightings per utterance. During inference, mixed emotional speech can be generated by flexibly adjusting emotion proportions and leveraging LLM contextual knowledge, enabling the model to quantify different emotional styles. Our proposed PUE successfully facilitates expressive speech synthesis of unseen emotions in a zero-shot setting.