mlsanityData to 7 Oct 2026

Differentiable Reward Optimization for LLM based TTS system

arXiv 2507.05911 ↗First posted 8 Jul 2025cs.SD cs.AI eess.AScited by 15

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

This paper proposes a novel Differentiable Reward Optimization (DiffRO) method aimed at enhancing the performance of neural codec language models based text-to-speech (TTS) systems. In contrast to conventional reinforcement learning from human feedback (RLHF) approaches applied to TTS, DiffRO directly compute the rewards based on neural codec tokens, rather than relying on synthesized audio. Furthermore, we employ the Gumbel-Softmax technique to render the reward function differentiable, thereby streamlining the RLHF training process. Additionally, we introduce a multi-task reward (MTR) model which can provide feedback from different perspectives and find that it can augment the system's capability to follow instructions effectively.Experimental results indicate that DiffRO significantly improves the pronunciation accuracy of the TTS system, achieving state-of-the-art (SOTA) WER results on the seed-tts-eval benchmark. Moreover, with the integration of the MTR model, we demonstrate the ability to control emotional and quality attributes in a zero-shot manner.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 27 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

25 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization2 Jul
  2. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback2 Jun
  3. 2024BATON: Aligning Text-to-Audio Model with Human Preference Feedback1 Feb

Other references · 22

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  9. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  10. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  11. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  12. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
10 more
  1. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  2. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 165
  3. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
  4. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  5. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  6. 2022The VoiceMOS Challenge 2022cited by 57
  7. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 56
  8. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 39
  9. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 31
  10. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2026FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learningcited by 5
  2. 2025RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTScited by 4
  3. 2025No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTScited by 3
  4. 2025Explore the Reinforcement Learning for the LLM based ASR and TTS systemcited by 3
  5. 2026FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speechcited by 3
  6. 2025Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesiscited by 2
  7. 2026HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speechcited by 1
  8. 2026GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech modelcited by 1
  9. 2025GRADE: Replacing Policy Gradients with Backpropagation for LLM Alignmentcited by 0
  10. 2026Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generationcited by 0
  11. 2026On the Emotion Understanding of Synthesized Speechcited by 0
  12. 2026Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignmentcited by 0
3 more
  1. 2026End-to-End Training for Discrete Token LLM based TTS Systemcited by 0
  2. 2026When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Modelscited by 0
  3. 2026NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
  2. 2024Preference Alignment Improves Language Model-Based TTS
  3. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimization
  4. 2026Post-Training Speech Enhancement Language Models with Perceptual Rewards
  5. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization
  6. 2025GLM-TTS Technical Report
  7. 2024T-REG: Preference Optimization with Token-Level Reward Regularization
  8. 2025Fine-Tuning Text-to-Speech Diffusion Models Using Reinforcement Learning with Human Feedback