What Did You Say? Task-Oriented Dialog Datasets Are Not Conversational!?
Not on the map. Why?
Abstract (arXiv)
High-quality datasets for task-oriented dialog are crucial for the development of virtual assistants. Yet three of the most relevant large scale dialog datasets suffer from one common flaw: the dialog state update can be tracked, to a great extent, by a model that only considers the current user utterance, ignoring the dialog history. In this work, we outline a taxonomy of conversational and contextual effects, which we use to examine MultiWOZ, SGD and SMCalFlow, among the most recent and widely used task-oriented dialog datasets. We analyze the datasets in a model-independent fashion and corroborate these findings experimentally using a strong text-to-text baseline (T5). We find that less than 4% of MultiWOZ's turns and 10% of SGD's turns are conversational, while SMCalFlow is not conversational at all in its current release: its dialog state tracking task can be reduced to single exchange semantic parsing. We conclude by outlining desiderata for truly conversational dialog datasets.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 7 Mar 2022.
Builds on
23 of 27 references.
References · 23
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
- 2021Carbon Emissions and Large Neural Network Trainingcited by 561
- 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
- 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
- 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
- 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
- 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
- 2015A Neural Network Approach to Context-Sensitive Generation of Conversational Responsescited by 98
- 2021MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluationcited by 86
- 2019Dialog State Tracking: A Neural Reading Comprehension Approachcited by 62
11 more
- 2019Improving Multi-turn Dialogue Modelling with Utterance ReWritercited by 57
- 2020A Sequence-to-Sequence Approach to Dialogue State Trackingcited by 43
- 2017Hierarchical Recurrent Attention Network for Response Generationcited by 34
- 2019ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generationcited by 32
- 2021Slot Self-Attentive Dialogue State Trackingcited by 27
- 2021Effective Sequence-to-Sequence Dialogue State Trackingcited by 20
- 2019Modeling Semantic Relationship in Multi-turn Conversations with Hierarchical Latent Variablescited by 19
- 2020Structured Attention for Unsupervised Dialogue Structure Inductioncited by 19
- 2018Sequence-to-Sequence Learning for Task-oriented Dialogue with Dialogue State Representationcited by 16
- 2020Semantic Role Labeling Guided Multi-turn Dialogue ReWritercited by 15
- 2020Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Trackingcited by 11
Built on by
Citing papers per year: 2022: 2.
2 papers.