mlsanityData to 14 Oct 2026

Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback

arXiv 2411.01834 ↗First posted 4 Nov 2024cs.CL eess.AScited by 23

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

While textless Spoken Language Models (SLMs) have shown potential in end-to-end speech-to-speech modeling, they still lag behind text-based Large Language Models (LLMs) in terms of semantic coherence and relevance. This work introduces the Align-SLM framework, which leverages preference optimization inspired by Reinforcement Learning with AI Feedback (RLAIF) to enhance the semantic understanding of SLMs. Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO). We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation. Experimental results show that our method achieves state-of-the-art performance for SLMs on most benchmarks, highlighting the importance of preference optimization to improve the semantics of SLMs.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 53 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

41 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
28 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  5. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  6. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  7. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  8. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  9. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  10. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  11. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  12. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  13. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  14. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  15. 2021Generalization Ability of MOS Prediction Networkscited by 69
  16. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  17. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  18. 2024Scaling Properties of Speech Language Modelscited by 32
  19. 2023Toward Joint Language Modeling for Speech Units and Textcited by 28
  20. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  21. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  22. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  23. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  24. 2024BATON: Aligning Text-to-Audio Model with Human Preference Feedbackcited by 18
  25. 2022DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answeringcited by 15
  26. 2023GSQA: An End-to-End Model for Generative Spoken Question Answeringcited by 9
  27. 2023Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Targetcited by 6
  28. 2024Can LLMs Understand the Implication of Emphasized Sentences in Dialogue?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 8.

23 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  4. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  5. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 14
  6. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Modelscited by 13
  7. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 12
  8. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  9. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 10
  10. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8
  11. 2026Covo-Audio Technical Reportcited by 7
  12. 2025SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Modelscited by 5
11 more
  1. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  2. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  3. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  4. 2025EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systemscited by 2
  5. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesiscited by 1
  6. 2026Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Modelscited by 1
  7. 2025GmSLM : Generative Marmoset Spoken Language Modelingcited by 0
  8. 2025Cross-Lingual Interleaving for Speech Language Modelscited by 0
  9. 2026AP-GRPO: Anchor-Gated Phonetic Alignment with Policy Optimization for Pathological Speech Reconstructioncited by 0
  10. 2026On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Modelscited by 0
  11. 2026Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  3. 2024Preference Alignment Improves Language Model-Based TTS
  4. 2024SpeechAlign: Aligning Speech Generation to Human Preferences
  5. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  6. 2024Investigating on RLHF methodology
  7. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  8. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment