mlsanityData to 7 Oct 2026

Aligning Spoken Dialogue Models from User Interactions

arXiv 2506.21463 ↗First posted 26 Jun 2025cs.CL cs.LG cs.SDcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

We propose a novel preference alignment framework for improving spoken dialogue models on real-time conversations from user interactions. Current preference learning methods primarily focus on text-based language models, and are not directly suited to the complexities of real-time speech interactions, with richer dynamics (e.g. interruption, interjection) and no explicit segmentation between speaker turns.We create a large-scale dataset of more than 150,000 preference pairs from raw multi-turn speech conversations, annotated with AI feedback, to cover preferences over both linguistic content and temporal context variations. We leverage offline alignment methods to finetune a full-duplex autoregressive speech-to-speech model. Extensive experiments demonstrate that feedback on generic conversations can be consistently effective in improving spoken dialogue models to produce more factual, safer and more contextually aligned interactions. We deploy the finetuned model and conduct holistic human evaluations to assess the impact beyond single-turn conversations. Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 46 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

39 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Preference Alignment Improves Language Model-Based TTS19 Sep
  2. 2024SpeechAlign: Aligning Speech Generation to Human Preferences8 Apr
  3. 2024MusicRL: Aligning Music Generation to Human Preferences6 Feb

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
24 more
  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  4. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  9. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  10. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  11. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  12. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  13. 2022Generative Spoken Dialogue Language Modelingcited by 125
  14. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  15. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  16. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  17. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  18. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 45
  19. 2024A Full-duplex Speech Dialogue Scheme Based On Large Language Modelscited by 40
  20. 2024Voice Jailbreak Attacks Against GPT-4ocited by 39
  21. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  22. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 31
  23. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  24. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  2. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 9
  3. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  4. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  5. 2026WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Trainingcited by 3
  6. 2025TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleavingcited by 1
  7. 2026Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Modelscited by 1
  8. 2026Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimizationcited by 0
  9. 2026NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translationcited by 0
  10. 2026RetroThinker: Enabling Retrospective Thinking in Speech LLMscited by 0
  11. 2026SteerDuplex: Steerable Duplex Speech Dialogue Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
  2. 2025Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment
  4. 2026SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness
  5. 2025Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction
  6. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models
  7. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
  8. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback