mlsanityData to 7 Oct 2026

Preference Alignment Improves Language Model-Based TTS

arXiv 2409.12403 ↗First posted 19 Sep 2024cs.CL cs.AIcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Recent advancements in text-to-speech (TTS) have shown that language model (LM)-based systems offer competitive performance to their counterparts. Further optimization can be achieved through preference alignment algorithms, which adjust LMs to align with the preferences of reward models, enhancing the desirability of the generated content. This study presents a thorough empirical evaluation of how preference alignment algorithms, particularly Direct Preference Optimization (DPO), enhance LM-based TTS. With a 1.15B parameter LM-based TTS model, we demonstrate that preference alignment consistently improves intelligibility, speaker similarity, and proxy subjective evaluation scores, with the latter two metrics surpassing even human speech in certain evaluations. We also show preference alignment is applicable to low-resource scenarios and effectively generalized to out-of-domain applications.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 55 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

44 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization2 Jul
  2. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback2 Jun
  3. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization15 Apr
  4. 2024SpeechAlign: Aligning Speech Generation to Human Preferences8 Apr

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  11. 2022High Fidelity Neural Audio Compressioncited by 673
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
28 more
  1. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  2. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  3. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  4. 2023Simple and Controllable Music Generationcited by 480
  5. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  6. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  8. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  9. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  10. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  11. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  12. 2021A Survey on Neural Speech Synthesiscited by 182
  13. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  14. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  15. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  16. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  17. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  18. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 62
  19. 2021Lower Perplexity is Not Always Human-Likecited by 52
  20. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  21. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  22. 2024Towards audio language modeling -- an overviewcited by 40
  23. 2019RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verificationcited by 33
  24. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  25. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  26. 2023AutoPrep: An Automatic Preprocessing Framework for In-the-Wild Speech Datacited by 16
  27. 2024The Interspeech 2024 Challenge on Speech Processing Using Discrete Unitscited by 12
  28. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 6.

21 papers.

Sort:

By area:

  1. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 26
  2. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  3. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  4. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  5. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  6. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  7. 2025Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Modelcited by 7
  8. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 7
  9. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  10. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  11. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Modelscited by 4
  12. 2025RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTScited by 4
9 more
  1. 2026FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speechcited by 3
  2. 2026Preference Optimization for Non-Verbal Vocalization Synthesiscited by 2
  3. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluatorcited by 1
  4. 2025Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Rewardcited by 1
  5. 2026Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speechcited by 1
  6. 2025Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speechcited by 0
  7. 2025Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Modelscited by 0
  8. 2026Imitation Learning for Elder-Facing Speech Synthesiscited by 0
  9. 2026Ready to Speak: Aligning LLMs for TTS-Friendly Text Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  2. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  3. 2025ComPO: Preference Alignment via Comparison Oracles
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  6. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  7. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  8. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization