mlsanityData to 7 Oct 2026

No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS

arXiv 2509.18531 ↗First posted 23 Sep 2025eess.AS cs.AI cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Recent work reports gains in neural text-to-speech (TTS) with Group Relative Policy Optimization (GRPO). However, in the absence of a verifiable reward for textitprosody, GRPO trained on transcription-oriented signals (CER/NLL) lowers error rates yet collapses prosody into monotone, unnatural speech; adding speaker-similarity further destabilizes training and degrades CER. We address this with an textititerative Direct Preference Optimization (DPO) scheme that uses only a few hundred human-labeled preference pairs per round to directly optimize prosodic naturalness while regularizing to the current model. On KoCC-TTS, a curated dataset of authentic Korean call center interactions capturing task-oriented dialogues, our method attains the highest human preference (ELO) with competitive CER, outperforming GRPO and strong commercial baselines. These results suggest that when prosody cannot be rewarded automatically, textithuman preference optimization offers a practical and data-efficient path to natural and robust TTS. The demo page is available at hrefhttps://tts.ch.dev

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 19 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

15 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Differentiable Reward Optimization for LLM based TTS system8 Jul
  2. 2024SpeechAlign: Aligning Speech Generation to Human Preferences8 Apr

Other references · 13

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  6. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  7. 2024Iterative Reasoning Preference Optimizationcited by 244
  8. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  9. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  10. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 31
  11. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  12. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 7
1 more
  1. 2025TTS-1 Technical Reportcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026GLASS: GRPO-Trained LoRA for Acoustic Style Steering in Zero-Shot Text-to-Speechcited by 1
  2. 2026HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speechcited by 1
  3. 2026The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speechcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
  2. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization
  3. 2026Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
  4. 2024Preference Alignment Improves Language Model-Based TTS
  5. 2025Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech
  6. 2026SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness
  7. 2026EmphTTS: an emphasis-control TTS with reinforcement learning
  8. 2026Direct Preference Density Alignment for Conversational Audio Equalization