mlsanityData to 7 Oct 2026

SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents

arXiv 2305.13040 ↗First posted 22 May 2023cs.CL cs.AIcited by 71

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Task-oriented dialogue (TOD) models have made significant progress in recent years. However, previous studies primarily focus on datasets written by annotators, which has resulted in a gap between academic research and real-world spoken conversation scenarios. While several small-scale spoken TOD datasets are proposed to address robustness issues such as ASR errors, they ignore the unique challenges in spoken conversation. To tackle the limitations, we introduce SpokenWOZ, a large-scale speech-text dataset for spoken TOD, containing 8 domains, 203k turns, 5.7k dialogues and 249 hours of audios from human-to-human spoken conversations. SpokenWOZ further incorporates common spoken characteristics such as word-by-word processing and reasoning in spoken language. Based on these characteristics, we present cross-turn slot and reasoning slot detection as new challenges. We conduct experiments on various baselines, including text-modal models, newly proposed dual-modal models, and LLMs, e.g., ChatGPT. The results show that the current models still have substantial room for improvement in spoken conversation, where the most advanced dialogue state tracker only achieves 25.65% in joint goal accuracy and the SOTA end-to-end model only correctly completes the user request in 52.1% of dialogues. The dataset, code, and leaderboard are available: https://spokenwoz.github.io/.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 24 of its 45 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

24 of 45 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  7. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  8. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  9. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  10. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  11. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  12. 2017Key-Value Retrieval Networks for Task-Oriented Dialoguecited by 137
12 more
  1. 2020TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Trackingcited by 130
  2. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  3. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  4. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  5. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  6. 2018Building a Conversational Agent Overnight with Dialogue Self-Playcited by 70
  7. 2020RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systemscited by 36
  8. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  9. 2022A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slotscited by 13
  10. 2022SCL-RAI: Span-based Contrastive Learning with Retrieval Augmented Inference for Unlabeled Entity Problem in NERcited by 11
  11. 2022CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluationcited by 9
  12. 2022EVI: Multilingual Spoken Dialogue Tasks and Dataset for Knowledge-Based Enrolment, Verification, and Identificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 26; 2026: 24.

71 papers.

Sort:

By area:

  1. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  2. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  3. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  4. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  5. 2025LLM Agents for Education: Advances and Applicationscited by 59
  6. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  7. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  8. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domainscited by 30
  9. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 20
  10. 2026From Context to Skills: Can Language Models Learn from Context Skillfully?cited by 19
  11. 2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 18
  12. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agentscited by 17
38 more
  1. 2025A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Taskscited by 17
  2. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  3. 2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
  4. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
  5. 2025Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filteringcited by 12
  6. 2025InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-trainingcited by 11
  7. 2023Mining Clues from Incomplete Utterance: A Query-enhanced Network for Incomplete Utterance Rewritingcited by 9
  8. 2025OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarioscited by 9
  9. 2025MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Modelscited by 9
  10. 2024Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extractioncited by 8
  11. 2025DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Datasetcited by 8
  12. 2026Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputscited by 8
  13. 2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 7
  14. 2025The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogscited by 6
  15. 2023Self-Explanation Prompting Improves Dialogue Understanding in Large Language Modelscited by 5
  16. 2025AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Taskscited by 5
  17. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 5
  18. 2024Chitchat as Interference: Adding User Backstories to Task-Oriented Dialoguescited by 4
  19. 2024Dual-view X-ray Detection: Can AI Detect Prohibited Items from Dual-view X-ray Images like Humans?cited by 4
  20. 2025Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMscited by 4
  21. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  22. 2026EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptionscited by 4
  23. 2023UniPCM: Universal Pre-trained Conversation Model with Task-aware Automatic Promptcited by 3
  24. 2025From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decompositioncited by 3
  25. 2026Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptionscited by 3
  26. 2023Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learningcited by 2
  27. 2025Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extractioncited by 2
  28. 2025Dialogue Response Prefetching Based on Semantic Similarity and Prediction Confidence of Language Modelcited by 2
  29. 2026MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homescited by 2
  30. 2026Hear2Act: Benchmarking When Prosody Should Change What an Assistant Doescited by 2
  31. 2026Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Modelscited by 2
  32. 2023Is one brick enough to break the wall of spoken dialogue state tracking?cited by 1
  33. 2024Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognitioncited by 1
  34. 2024Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillationcited by 1
  35. 2024Schema Augmentation for Zero-Shot Domain Adaptation in Dialogue State Trackingcited by 1
  36. 2025The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approachcited by 1
  37. 2025Proactive Hearing Assistants that Isolate Egocentric Conversationscited by 1
  38. 2025Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Trackingcited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models
  2. 2024Large Language Models as Zero-shot Dialogue State Tracker through Function Calling
  3. 2024Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents
  4. 2025Aligning Spoken Dialogue Models from User Interactions
  5. 2026Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue
  6. 2024Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models
  7. 2023LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
  8. 2025Spoken Conversational Agents with Large Language Models