Sources of Noise in Dialogue and How to Deal with Them
Not on the map. Why?
Abstract (arXiv)
Training dialogue systems often entails dealing with noisy training examples and unexpected user inputs. Despite their prevalence, there currently lacks an accurate survey of dialogue noise, nor is there a clear sense of the impact of each noise type on task performance. This paper addresses this gap by first constructing a taxonomy of noise encountered by dialogue systems. In addition, we run a series of experiments to show how different models behave when subjected to varying levels of noise and types of noise. Our results reveal that models are quite robust to label errors commonly tackled by existing denoising algorithms, but that performance suffers from dialogue-specific noise. Driven by these observations, we design a data cleaning algorithm specialized for conversational settings and apply it as a proof-of-concept for targeted dialogue denoising.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 47 of its 91 references that are in the map. First posted on arXiv 6 Dec 2022.
Builds on
47 of 91 references.
References · 47
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
- 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
- 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
35 more
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
- 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
- 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
- 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
- 2021A Survey of Data Augmentation Approaches for NLPcited by 408
- 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
- 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
- 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
- 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
- 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
- 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
- 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
- 2017Key-Value Retrieval Networks for Task-Oriented Dialoguecited by 137
- 2018Dialogue Natural Language Inferencecited by 135
- 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
- 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
- 2020SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustnesscited by 94
- 2021MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluationcited by 86
- 2017Who Said What: Modeling Individual Labelers Improves Classificationcited by 77
- 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
- 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
- 2020MultiWOZ 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotationcited by 51
- 2017Learning From Noisy Singly-labeled Datacited by 50
- 2019Domain Adaptive Dialog Generation via Meta Learningcited by 41
- 2021SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failurescited by 37
- 2020RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systemscited by 36
- 2020Robustness Testing of Language Understanding in Task-Oriented Dialogcited by 32
- 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
- 2021Did they answer? Subjective acts and intents in conversational discoursecited by 22
- 2019MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversationscited by 20
- 2021GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentationcited by 20
- 2019Learning with Noisy Labels for Sentence-level Sentiment Classificationcited by 11
- 2021HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversationscited by 9
- 2018Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervisioncited by 6
- 2022Data Augmentation for Intent Classificationcited by 3
Built on by
Citing papers per year: 2023: 1.
1 paper.