mlsanityData to 14 Oct 2026

SpeechAlign: Aligning Speech Generation to Human Preferences

arXiv 2404.05600 ↗First posted 8 Apr 2024cs.CL cs.SD eess.AScited by 47

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Speech language models have significantly advanced in generating realistic speech, with neural codec language models standing out. However, the integration of human feedback to align speech outputs to human preferences is often neglected. This paper addresses this gap by first analyzing the distribution gap in codec language models, highlighting how it leads to discrepancies between the training and inference phases, which negatively affects performance. Then we explore leveraging learning from human feedback to bridge the distribution gap. We introduce SpeechAlign, an iterative self-improvement strategy that aligns speech language models to human preferences. SpeechAlign involves constructing a preference codec dataset contrasting golden codec tokens against synthetic tokens, followed by preference optimization to improve the codec language model. This cycle of improvement is carried out iteratively to steadily convert weak models to strong ones. Through both subjective and objective evaluations, we show that SpeechAlign can bridge the distribution gap and facilitating continuous self-improvement of the speech language model. Moreover, SpeechAlign exhibits robust generalization capabilities and works for smaller models. Code and models will be available at https://github.com/0nutation/SpeechGPT.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 22 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

16 of 22 references.

References · 16

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  9. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  10. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  11. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  12. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
4 more
  1. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  2. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  3. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  4. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solutioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 17; 2026: 19.

47 papers.

Sort:

By area:

  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  2. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  4. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
  5. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  6. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  7. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  8. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  9. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  10. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 11
  11. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  12. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 10
35 more
  1. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 8
  2. 2024Takin: A Cohort of Superior Quality Zero-shot Speech Generation Modelscited by 7
  3. 2025Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Modelcited by 7
  4. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  5. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  6. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  7. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  8. 2026CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instructioncited by 5
  9. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  10. 2025No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTScited by 4
  11. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Modelscited by 4
  12. 2024Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representationcited by 3
  13. 2026WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Trainingcited by 3
  14. 2026TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosiscited by 2
  15. 2026Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMscited by 2
  16. 2026The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speechcited by 2
  17. 2025Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generationcited by 1
  18. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimizationcited by 1
  19. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesiscited by 1
  20. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluatorcited by 1
  21. 2025Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Rewardcited by 1
  22. 2026AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Stylecited by 1
  23. 2026Post-Training Speech Enhancement Language Models with Perceptual Rewardscited by 1
  24. 2026Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correctioncited by 1
  25. 2025On Monotonicity in AI Alignmentcited by 0
  26. 2025Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speechcited by 0
  27. 2026Same Words, Different Judgments: How Preferences Vary Across Modalitiescited by 0
  28. 2026Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interactioncited by 0
  29. 2026AP-GRPO: Anchor-Gated Phonetic Alignment with Policy Optimization for Pathological Speech Reconstructioncited by 0
  30. 2026Imitation Learning for Elder-Facing Speech Synthesiscited by 0
  31. 2026Fr'echet Distance Loss on Speech Representations for Text-to-Speech Synthesiscited by 0
  32. 2026Best-of-N TTS Evaluation is Confounded by ASR Family Alignmentcited by 0
  33. 2026When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Modelscited by 0
  34. 2026EmphTTS: an emphasis-control TTS with reinforcement learningcited by 0
  35. 2026Can LLM Agents Automate Reinforcement Learning for Text-to-Speech?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Aligning Large Language Models through Synthetic Feedback
  2. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models
  3. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  4. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  5. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  6. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  7. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  8. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment