Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?
Not on the map. Why?
Abstract (arXiv)
Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.
Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 15 May 2026.
Builds on
21 of 31 references.
References · 21
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,366
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 550
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 498
- 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 468
- 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 335
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 284
- 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
- 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
- 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
- 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 137
- 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 126
9 more
- 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
- 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 59
- 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 53
- 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
- 2025Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examinercited by 38
- 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 20
- 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 20
- 2025ContextASR-Bench: A Massive Contextual Speech Recognition Benchmarkcited by 14
- 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 14
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflowscited by 5
- 2026MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agentscited by 3
- 2026DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agentscited by 1
- 2026Full-Duplex Speech Models Take the Floor When Asked, Not When Neededcited by 1
- 2026Coupled but Late: Turn-Taking Between Full-Duplex Speech Models in Unscripted Dialoguecited by 0
- 2026Coupled but Late: Turn-Taking Between Full-Duplex Speech Models in Unscripted Dialogue6 Oct 2026
- 2026MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents21 Sep 2026
- 2026Full-Duplex Speech Models Take the Floor When Asked, Not When Needed17 Sep 2026
- 2026DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents3 Sep 2026
- 2026IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows17 Jun 2026