mlsanityData to 13 Oct 2026

Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?

arXiv 2607.20460 ↗First posted 15 May 2026cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

21 of 31 references.

References · 21

  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,366
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 550
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 498
  5. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 468
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 335
  7. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 284
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  9. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  10. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  11. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 137
  12. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 126
9 more
  1. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  2. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 59
  3. 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 53
  4. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  5. 2025Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examinercited by 38
  6. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 20
  7. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 20
  8. 2025ContextASR-Bench: A Massive Contextual Speech Recognition Benchmarkcited by 14
  9. 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 5.

5 papers.