# HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech

arXiv: https://arxiv.org/abs/2606.28249 · HTML: https://mlsanity.com/paper/2606.28249/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot](https://mlsanity.com/paper/2509.02020/) (2025)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization](https://mlsanity.com/paper/2409.10157/) (2024)
- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability](https://mlsanity.com/paper/2104.01408/) (2021)
- [LSSED: a large-scale dataset and benchmark for speech emotion recognition](https://mlsanity.com/paper/2102.01754/) (2021)
- [Differentiable Reward Optimization for LLM based TTS system](https://mlsanity.com/paper/2507.05911/) (2025)
- [GLM-TTS Technical Report](https://mlsanity.com/paper/2512.14291/) (2025)
- [Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis](https://mlsanity.com/paper/2509.24629/) (2025)
- [No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS](https://mlsanity.com/paper/2509.18531/) (2025)
- [HD-PPT: Hierarchical Decoding of Content- and Prompt-Preference Tokens for Instruction-based TTS](https://mlsanity.com/paper/2509.19001/) (2025)
- [EMORL-TTS: Reinforcement Learning for Fine-Grained Emotion Control in LLM-based TTS](https://mlsanity.com/paper/2510.05758/) (2025)
- [RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS](https://mlsanity.com/paper/2512.04552/) (2025)
- [Beyond Global Emotion: Fine-Grained Emotional Speech Synthesis with Dynamic Word-Level Modulation](https://mlsanity.com/paper/2509.20378/) (2025)
- [Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Reward](https://mlsanity.com/paper/2511.17555/) (2025)
- [Emo-LiPO: Listwise Preference Optimization for Fine-Grained Emotion Intensity Control in LLM-based Text-to-Speech](https://mlsanity.com/paper/2606.13006/) (2026)

## Built on by (most cited first, 1 of 1)

- [WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS](https://mlsanity.com/paper/2607.06461/) (2026)

## Abstract (arXiv)

Recently, Large Language Model (LLM)-based Text-to-Speech (TTS) models have achieved remarkable naturalness. However, the standard Supervised Fine-Tuning paradigm often converges to statistically averaged prosody, limiting emotional expressiveness. While preference-driven optimization offers a promising alternative, existing approaches suffer from two structural mismatches: information conflict, where content and emotion in a shared latent space produce conflicting gradients, leading to reward hacking and semantic degradation; and scale gap, where sparse sentence-level rewards struggle to guide dense frame-level generation. To overcome these challenges, we propose HPRO, a hierarchical progressive reward optimization framework. Within HPRO, we introduce the HD-Emo codec as a novel differentiable reward model to mitigate the information conflict. It extracts speech into distinct content and style preference tokens, structurally isolating emotional optimization from semantic content. Building upon this structured preference space, HPRO bridges the scale gap by progressively aligning frame-, word- and sentence-level objectives. Experiments demonstrate that HPRO significantly enhances emotional expressiveness, while effectively preserving linguistic intelligibility. The code and audio samples are publicly available at https://xxh333.github.io/hpro-demo/.