# Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models

arXiv: https://arxiv.org/abs/2604.14920 · HTML: https://mlsanity.com/paper/2604.14920/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 22 of its 82 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 22 of 22)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis](https://mlsanity.com/paper/2505.02625/) (2025)
- [Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents](https://mlsanity.com/paper/2409.15594/) (2024)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks](https://mlsanity.com/paper/2409.13832/) (2024)
- [SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation](https://mlsanity.com/paper/2505.17060/) (2025)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2411.01834/) (2024)
- [NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction](https://mlsanity.com/paper/2506.00975/) (2025)
- [WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models](https://mlsanity.com/paper/2502.14727/) (2025)
- [Audio-Aware Large Language Models as Judges for Speaking Styles](https://mlsanity.com/paper/2506.05984/) (2025)
- [WavReward: Spoken Dialogue Models With Generalist Reward Evaluators](https://mlsanity.com/paper/2505.09558/) (2025)
- [Aligning Spoken Dialogue Models from User Interactions](https://mlsanity.com/paper/2506.21463/) (2025)
- [WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Models](https://mlsanity.com/paper/2602.12135/) (2026)
- [Evaluating Speech-to-Text x LLM x Text-to-Speech Combinations for AI Interview Systems](https://mlsanity.com/paper/2507.16835/) (2025)

## Built on by (most cited first, 1 of 1)

- [Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models](https://mlsanity.com/paper/2606.11167/) (2026)

## Abstract (arXiv)

Achieving seamless, human-like interaction remains a key challenge for full-duplex spoken dialogue models (SDMs). Reinforcement learning (RL) has substantially enhanced text- and vision-language models, while well-designed reward signals are crucial for the performance of RL. We consider RL a promising strategy to address the key challenge for SDMs. However, a fundamental barrier persists: prevailing automated metrics for assessing interaction quality rely on superficial proxies, such as behavioral statistics or timing-prediction accuracy, failing to provide reliable reward signals for RL. On the other hand, human evaluations, despite their richness, remain costly, inconsistent, and difficult to scale. We tackle this critical barrier by proposing a Dual-Axis Generative Reward Model, which is trained to understand complex interaction dynamics using a detailed taxonomy and an annotated dataset, produces a single score and, crucially, provides separate evaluations for semantic quality and interaction timing. Such dual outputs furnish precise diagnostic feedback for SDMs and deliver a dependable, instructive reward signal suitable for online reinforcement learning. Our model achieves state-of-the-art performance on interaction-quality assessment across a wide spectrum of datasets, spanning synthetic dialogues and complex real-world interactions.