# MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation

arXiv: https://arxiv.org/abs/2104.00773 · HTML: https://mlsanity.com/paper/2104.00773/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking](https://mlsanity.com/paper/2005.02877/) (2020)
- [A Survey of Label-noise Representation Learning: Past, Present and Future](https://mlsanity.com/paper/2011.04406/) (2020)
- [Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/1704.00057/) (2017)
- [Efficient Dialogue State Tracking by Selectively Overwriting Memory](https://mlsanity.com/paper/1911.03906/) (2019)
- [Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/1910.03544/) (2019)
- [CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset](https://mlsanity.com/paper/2002.11893/) (2020)
- [SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking](https://mlsanity.com/paper/1907.07421/) (2019)
- [MultiWOZ 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation](https://mlsanity.com/paper/2010.05594/) (2020)
- [Global-Locally Self-Attentive Dialogue State Tracker](https://mlsanity.com/paper/1805.09655/) (2018)
- [CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers](https://mlsanity.com/paper/2010.12850/) (2020)
- [RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modeling](https://mlsanity.com/paper/2010.08738/) (2020)
- [Conversational Semantic Parsing for Dialog State Tracking](https://mlsanity.com/paper/2010.12770/) (2020)
- [Slot Self-Attentive Dialogue State Tracking](https://mlsanity.com/paper/2101.09374/) (2021)
- [A Contextual Hierarchical Attention Network with Adaptive Objective for Dialogue State Tracking](https://mlsanity.com/paper/2006.01554/) (2020)
- [Parallel Interactive Networks for Multi-Domain Dialogue State Generation](https://mlsanity.com/paper/2009.07616/) (2020)

## Built on by (most cited first, 30 of 88)

- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [Description-Driven Task-Oriented Dialog Modeling](https://mlsanity.com/paper/2201.08904/) (2022)
- [In-Context Learning for Few-Shot Dialogue State Tracking](https://mlsanity.com/paper/2203.08568/) (2022)
- [API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs](https://mlsanity.com/paper/2402.15491/) (2024)
- [A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding](https://mlsanity.com/paper/2304.04256/) (2023)
- [Fusing task-oriented and open-domain dialogues in conversational agents](https://mlsanity.com/paper/2109.04137/) (2021)
- [Towards LLM-driven Dialogue State Tracking](https://mlsanity.com/paper/2310.14970/) (2023)
- [Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation](https://mlsanity.com/paper/2405.13037/) (2024)
- ["Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking](https://mlsanity.com/paper/2207.14627/) (2022)
- [Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2203.01552/) (2022)
- [Goal Alignment in LLM-Based User Simulators for Conversational AI](https://mlsanity.com/paper/2507.20152/) (2025)
- [Effective Sequence-to-Sequence Dialogue State Tracking](https://mlsanity.com/paper/2108.13990/) (2021)
- [Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model](https://mlsanity.com/paper/2502.08820/) (2025)
- [OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Tracking](https://mlsanity.com/paper/2311.09758/) (2023)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation](https://mlsanity.com/paper/2404.09682/) (2024)
- [Robust Dialogue State Tracking with Weak Supervision and Sparse Data](https://mlsanity.com/paper/2202.03354/) (2022)
- [A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialog](https://mlsanity.com/paper/2207.13211/) (2022)
- [Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations](https://mlsanity.com/paper/2305.14556/) (2023)
- [Enhancing LLM Reliability via Explicit Knowledge Boundary Modeling](https://mlsanity.com/paper/2503.02233/) (2025)
- [MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversations](https://mlsanity.com/paper/2506.20100/) (2025)
- [Workflow Discovery from Dialogues in the Low Data Regime](https://mlsanity.com/paper/2205.11690/) (2022)
- [ASSIST: Towards Label Noise-Robust Dialogue State Tracking](https://mlsanity.com/paper/2202.13024/) (2022)
- [Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues](https://mlsanity.com/paper/2111.02574/) (2021)
- [Database Search Results Disambiguation for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2112.08351/) (2021)
- [Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking](https://mlsanity.com/paper/2307.01453/) (2023)
- [TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisons](https://mlsanity.com/paper/2504.19982/) (2025)
- [Frictional Agent Alignment Framework: Slow Down and Don't Break Things](https://mlsanity.com/paper/2505.19428/) (2025)

## Abstract (arXiv)

The MultiWOZ 2.0 dataset has greatly stimulated the research of task-oriented dialogue systems. However, its state annotations contain substantial noise, which hinders a proper evaluation of model performance. To address this issue, massive efforts were devoted to correcting the annotations. Three improved versions (i.e., MultiWOZ 2.1-2.3) have then been released. Nonetheless, there are still plenty of incorrect and inconsistent annotations. This work introduces MultiWOZ 2.4, which refines the annotations in the validation set and test set of MultiWOZ 2.1. The annotations in the training set remain unchanged (same as MultiWOZ 2.1) to elicit robust and noise-resilient model training. We benchmark eight state-of-the-art dialogue state tracking models on MultiWOZ 2.4. All of them demonstrate much higher performance than on MultiWOZ 2.1.