# Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization

arXiv: https://arxiv.org/abs/2409.10157 · HTML: https://mlsanity.com/paper/2409.10157/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2024.

## Built on by (most cited first, 30 of 32)

- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications](https://mlsanity.com/paper/2410.15595/) (2024)
- [F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimization](https://mlsanity.com/paper/2504.02407/) (2025)
- [Differentiable Reward Optimization for LLM based TTS system](https://mlsanity.com/paper/2507.05911/) (2025)
- [Aligning Spoken Dialogue Models from User Interactions](https://mlsanity.com/paper/2506.21463/) (2025)
- [OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis](https://mlsanity.com/paper/2501.04561/) (2025)
- [CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering](https://mlsanity.com/paper/2602.03420/) (2026)
- [Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model](https://mlsanity.com/paper/2506.08967/) (2025)
- [EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis](https://mlsanity.com/paper/2601.22873/) (2026)
- [Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models](https://mlsanity.com/paper/2409.18680/) (2024)
- [Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation](https://mlsanity.com/paper/2508.16332/) (2025)
- [FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech](https://mlsanity.com/paper/2606.23190/) (2026)
- [Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions](https://mlsanity.com/paper/2506.02742/) (2025)
- [No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS](https://mlsanity.com/paper/2509.18531/) (2025)
- [Direct Preference Optimization for Speech Autoregressive Diffusion Models](https://mlsanity.com/paper/2509.18928/) (2025)
- [UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement](https://mlsanity.com/paper/2510.20441/) (2025)
- [RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS](https://mlsanity.com/paper/2512.04552/) (2025)
- [A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models](https://mlsanity.com/paper/2607.00946/) (2026)
- [Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization](https://mlsanity.com/paper/2509.25416/) (2025)
- [HiStyle: Hierarchical Style Embedding Predictor for Text-Prompt-Guided Controllable Speech Synthesis](https://mlsanity.com/paper/2509.25842/) (2025)
- [TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosis](https://mlsanity.com/paper/2604.22225/) (2026)
- [Preference Optimization for Non-Verbal Vocalization Synthesis](https://mlsanity.com/paper/2608.24163/) (2026)
- [RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis](https://mlsanity.com/paper/2510.14628/) (2025)
- [The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation](https://mlsanity.com/paper/2604.26347/) (2026)
- [HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech](https://mlsanity.com/paper/2606.28249/) (2026)
- [Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models](https://mlsanity.com/paper/2510.13293/) (2025)
- [Same Words, Different Judgments: How Preferences Vary Across Modalities](https://mlsanity.com/paper/2602.22710/) (2026)
- [Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey](https://mlsanity.com/paper/2604.11283/) (2026)
- [Imitation Learning for Elder-Facing Speech Synthesis](https://mlsanity.com/paper/2606.21053/) (2026)

## Abstract (arXiv)

Current emotional text-to-speech (TTS) models predominantly conduct supervised training to learn the conversion from text and desired emotion to its emotional speech, focusing on a single emotion per text-speech pair. These models only learn the correct emotional outputs without fully comprehending other emotion characteristics, which limits their capabilities of capturing the nuances between different emotions. We propose a controllable Emo-DPO approach, which employs direct preference optimization to differentiate subtle emotional nuances between emotions through optimizing towards preferred emotions over less preferred emotional ones. Instead of relying on traditional neural architectures used in existing emotional TTS models, we propose utilizing the emotion-aware LLM-TTS neural architecture to leverage LLMs' in-context learning and instruction-following capabilities. Comprehensive experiments confirm that our proposed method outperforms the existing baselines.