# SteerDuplex: Steerable Duplex Speech Dialogue Models

arXiv: https://arxiv.org/abs/2609.12623 · HTML: https://mlsanity.com/paper/2609.12623/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 39 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Builds on (most cited first, 30 of 34)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization](https://mlsanity.com/paper/2601.05242/) (2026)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents](https://mlsanity.com/paper/2409.15594/) (2024)
- [PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models](https://mlsanity.com/paper/2602.06053/) (2026)
- [OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation](https://mlsanity.com/paper/2410.17799/) (2024)
- [Controllable Text Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2408.12599/) (2024)
- [Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models](https://mlsanity.com/paper/2507.23159/) (2025)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner](https://mlsanity.com/paper/2510.07838/) (2025)
- [A Full-duplex Speech Dialogue Scheme Based On Large Language Models](https://mlsanity.com/paper/2405.19487/) (2024)
- [Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2511.00222/) (2025)
- [SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model](https://mlsanity.com/paper/2505.15670/) (2025)
- [MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems](https://mlsanity.com/paper/2509.22047/) (2025)
- [AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation](https://mlsanity.com/paper/2507.12705/) (2025)
- [Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction](https://mlsanity.com/paper/2512.14865/) (2025)
- [Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers](https://mlsanity.com/paper/2606.12507/) (2026)
- [Aligning Spoken Dialogue Models from User Interactions](https://mlsanity.com/paper/2506.21463/) (2025)
- [Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR](https://mlsanity.com/paper/2605.20164/) (2026)
- [Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models](https://mlsanity.com/paper/2606.11167/) (2026)
- [MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions](https://mlsanity.com/paper/2507.10859/) (2025)

## Abstract (arXiv)

Full-duplex spoken dialogue models support low-latency turn taking, interruption handling, and backchanneling, yet a key capability remains underexplored: steerability, the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. We introduce a taxonomy of text- and audio-based steerability that identifies substantial gaps in current full-duplex models. To address this gap, we introduce SteerDuplex, a Moshi-based full-duplex speech model fine-tuned on natural conversations and synthetic dialogues targeting instruction following, vocal delivery, reasoning, and duplex interaction. We further apply two-stage reinforcement learning (RL) with hybrid rewards, combining verifiable interaction checks and judge-based semantic feedback to improve timing and response continuity. To evaluate full-duplex spoken steerability, we introduce SteerBench, a benchmark with 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style/accent, and speed/length. On SteerBench, supervised training improves audio-steering average pass rate by 44.5 percentage points over the strongest evaluated open baseline. On Audio MultiChallenge, task average pass rate improves by 7 points over its strongest evaluated open baseline. RL further raises source-clean interruption response from 72.5% to 82.5% and reduces synthetic pause barge-in from 26.5% to 9%. Steering and aggregate task scores remain comparable or higher, while reward probes reveal reward hacking through incomplete responses. Our model and benchmark support systematic research on spoken steerability, with reward analysis showing why timing gains must be evaluated alongside response completeness.