EmoInstruct-TTS: Dual-Path Instruction-Guided Emotional Speech Synthesis
Not on the map. Why?
Abstract (arXiv)
Instruction-based controllable speech synthesis enables users to specify emotions through natural language. However, existing approaches often rely on coarse emotion labels and lack explicit modeling of fine-grained intensity. We propose EmoInstruct-TTS, a dual-path instruction-guided framework for emotional speech synthesis. We introduce Emotion2embed, a supervised semantic-acoustic emotion embedding covering 48 emotional states, including fine-grained categories and intensity levels. To infer embeddings from free-form instructions, we design an Instruction-Conditioned Emotion Flow Model (ICE-Flow) that generates acoustically grounded emotion representations. The inferred embeddings are integrated into an LLM-based synthesis pipeline to provide explicit emotional control while preserving semantic planning. Experiments show improved emotional controllability and speech naturalness over strong baselines.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 8 Jun 2026.
Builds on
29 of 35 references.
References · 29
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
- 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
- 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
- 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
- 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
- 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
17 more
- 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
- 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
- 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 121
- 2026Qwen3-TTS Technical Reportcited by 118
- 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
- 2025MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encodercited by 48
- 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
- 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
- 2025FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbotcited by 35
- 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
- 2024EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Controlcited by 13
- 2026VoiceSculptor: Your Voice, Designed By Youcited by 10
- 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
- 2025InstructAudio: Unified speech and music generation with natural language instructioncited by 5
- 2024Description-based Controllable Text-to-Speech with Cross-Lingual Voice Controlcited by 4
- 2025EME-TTS: Unlocking the Emphasis and Emotion Link in Speech Synthesiscited by 2
- 2025MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Promptcited by 1
Built on by
Not cited yet.