mlsanityData to 7 Oct 2026

Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback

arXiv 2406.00654 ↗First posted 2 Jun 2024cs.CL cs.SD eess.AScited by 19

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

In recent years, text-to-speech (TTS) technology has witnessed impressive advancements, particularly with large-scale training datasets, showcasing human-level speech quality and impressive zero-shot capabilities on unseen speakers. However, despite human subjective evaluations, such as the mean opinion score (MOS), remaining the gold standard for assessing the quality of synthetic speech, even state-of-the-art TTS approaches have kept human feedback isolated from training that resulted in mismatched training objectives and evaluation metrics. In this work, we investigate a novel topic of integrating subjective human evaluation into the TTS training loop. Inspired by the recent success of reinforcement learning from human feedback, we propose a comprehensive sampling-annotating-learning framework tailored to TTS optimization, namely uncertainty-aware optimization (UNO). Specifically, UNO eliminates the need for a reward model or preference data by directly maximizing the utility of speech generations while considering the uncertainty that lies in the inherent variability in subjective human speech perception and evaluations. Experimental results of both subjective and objective evaluations demonstrate that UNO considerably improves the zero-shot performance of TTS models in terms of MOS, word error rate, and speaker similarity. Additionally, we present a remarkable ability of UNO that it can adapt to the desired speaking style in emotional TTS seamlessly and flexibly.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 75 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

59 of 75 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SpeechAlign: Aligning Speech Generation to Human Preferences8 Apr
  2. 2024MusicRL: Aligning Music Generation to Human Preferences6 Feb
  3. 2024BATON: Aligning Text-to-Audio Model with Human Preference Feedback1 Feb

Other references · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  11. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
38 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2024Self-Rewarding Language Modelscited by 654
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  10. 2023MusicLM: Generating Music From Textcited by 442
  11. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  12. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  13. 2019Deep Evidential Regressioncited by 354
  14. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  15. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  16. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  17. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  18. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  19. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  20. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  21. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  22. 2024Token-level Direct Preference Optimizationcited by 135
  23. 2024Direct Preference Optimization with an Offsetcited by 122
  24. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  25. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  26. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  27. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  28. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  29. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  30. 2021Generalization Ability of MOS Prediction Networkscited by 69
  31. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  32. 2023Better speech synthesis through scalingcited by 63
  33. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  34. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  35. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  36. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  37. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 33
  38. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 1.

19 papers.

Sort:

By area:

  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  2. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  3. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  4. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  5. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  6. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  7. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  8. 2024Takin: A Cohort of Superior Quality Zero-shot Speech Generation Modelscited by 7
  9. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 7
  10. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  11. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  12. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
7 more
  1. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  2. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Modelscited by 4
  3. 2025TTS-1 Technical Reportcited by 3
  4. 2025Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scalecited by 3
  5. 2025Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generationcited by 1
  6. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluatorcited by 1
  7. 2025Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Rewardcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
  2. 2025From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling
  3. 2026The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speech
  4. 2024Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR
  5. 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS
  6. 2026VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
  7. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  8. 2025Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech