Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework
Not on the map. Why?
Abstract (arXiv)
Compared with half-duplex dialogue systems where the system waits for user turn completion before it responds, natural full-duplex dialogue systems require agents to act proactively in real time, including timely interruptions and backchannels. This creates a key challenge: improving turn timing without sacrificing response quality. To address limitations in realistic proactive turn-taking, we build a generalized style-aware full-duplex framework with three key components. Firstly, we propose LPS-TC, a Lightweight Proactive Speech Turn Controller for plug-and-play integration. It features a fine-grained action space covering both reactive and proactive turn behaviors, enabling half-duplex models with full-duplex capabilities and enhancing existing full-duplex models with superior timing control. Secondly, we construct WildTurn, a large-scale, real-world English dataset containing approximately 2,981 hours of filtered multi-turn stereo conversations from face-to-face and telephone conversations, annotated with five turn-taking and five backchanneling styles. Trained on WildTurn, LPS-TC exhibits rich spoken dynamics that are not captured by existing static full-duplex benchmarks. Thirdly, we introduce a two-tier evaluation scheme that assesses both chunk-level timing precision and turn-level interaction quality under realistic streaming constraints. Our experiments, integrating LPS-TC with half-duplex models like Qwen2.5-Omni and full-duplex models like Freeze-Omni, showcase its superior performance in timing appropriateness and response quality. Our framework also demonstrates fine-grained style controllability and strong generalizability, enabling more natural and human-like spoken interactions.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 71 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
38 of 71 references.
References · 38
- 2025Qwen3 Technical Reportcited by 8,195
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2020Recipes for building an open-domain chatbotcited by 593
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
- 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
- 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
- 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
- 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
- 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
26 more
- 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
- 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
- 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 76
- 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
- 2023A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospectscited by 66
- 2024Language Model Can Listen While Speakingcited by 62
- 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
- 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 57
- 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 53
- 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
- 2025Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examinercited by 38
- 2025OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialoguecited by 21
- 2024Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusioncited by 20
- 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
- 2025NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Predictioncited by 19
- 2025FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementationscited by 17
- 2022Turn-Taking Prediction for Natural Conversational Speechcited by 15
- 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
- 2022UniDU: Towards A Unified Generative Dialogue Understanding Frameworkcited by 9
- 2025X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue Systemcited by 9
- 2025Open-Source Full-Duplex Conversational Datasets for Natural and Interactive Speech Synthesiscited by 3
- 2026Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorialcited by 3
- 2024Predicting and Understanding Turn-Taking Behavior in Open-Ended Group Activities in Virtual Realitycited by 2
- 2025CleanS2S: Single-file Framework for Proactive Speech-to-Speech Interactioncited by 2
- 2025Phoenix-VAD: Streaming Semantic Endpoint Detection for Full-Duplex Speech Interactioncited by 2
- 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2
Built on by
Not cited yet.