mlsanityData to 7 Oct 2026

Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework

arXiv 2608.28630 ↗First posted 4 Aug 2026cs.CL cs.AI cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Compared with half-duplex dialogue systems where the system waits for user turn completion before it responds, natural full-duplex dialogue systems require agents to act proactively in real time, including timely interruptions and backchannels. This creates a key challenge: improving turn timing without sacrificing response quality. To address limitations in realistic proactive turn-taking, we build a generalized style-aware full-duplex framework with three key components. Firstly, we propose LPS-TC, a Lightweight Proactive Speech Turn Controller for plug-and-play integration. It features a fine-grained action space covering both reactive and proactive turn behaviors, enabling half-duplex models with full-duplex capabilities and enhancing existing full-duplex models with superior timing control. Secondly, we construct WildTurn, a large-scale, real-world English dataset containing approximately 2,981 hours of filtered multi-turn stereo conversations from face-to-face and telephone conversations, annotated with five turn-taking and five backchanneling styles. Trained on WildTurn, LPS-TC exhibits rich spoken dynamics that are not captured by existing static full-duplex benchmarks. Thirdly, we introduce a two-tier evaluation scheme that assesses both chunk-level timing precision and turn-level interaction quality under realistic streaming constraints. Our experiments, integrating LPS-TC with half-duplex models like Qwen2.5-Omni and full-duplex models like Freeze-Omni, showcase its superior performance in timing appropriateness and response quality. Our framework also demonstrates fine-grained style controllability and strong generalizability, enabling more natural and human-like spoken interactions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 71 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

38 of 71 references.

References · 38

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  5. 2020Recipes for building an open-domain chatbotcited by 593
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  7. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  8. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  9. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  10. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  11. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  12. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
26 more
  1. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  2. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  3. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 76
  4. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
  5. 2023A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospectscited by 66
  6. 2024Language Model Can Listen While Speakingcited by 62
  7. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
  8. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 57
  9. 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 53
  10. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  11. 2025Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examinercited by 38
  12. 2025OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialoguecited by 21
  13. 2024Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusioncited by 20
  14. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  15. 2025NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Predictioncited by 19
  16. 2025FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementationscited by 17
  17. 2022Turn-Taking Prediction for Natural Conversational Speechcited by 15
  18. 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
  19. 2022UniDU: Towards A Unified Generative Dialogue Understanding Frameworkcited by 9
  20. 2025X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue Systemcited by 9
  21. 2025Open-Source Full-Duplex Conversational Datasets for Natural and Interactive Speech Synthesiscited by 3
  22. 2026Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorialcited by 3
  23. 2024Predicting and Understanding Turn-Taking Behavior in Open-Ended Group Activities in Virtual Realitycited by 2
  24. 2025CleanS2S: Single-file Framework for Proactive Speech-to-Speech Interactioncited by 2
  25. 2025Phoenix-VAD: Streaming Semantic Endpoint Detection for Full-Duplex Speech Interactioncited by 2
  26. 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2

Built on by

Not cited yet.