# Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering

arXiv: https://arxiv.org/abs/2205.10363 · HTML: https://mlsanity.com/paper/2205.10363/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 20 May 2022.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Key-Value Retrieval Networks for Task-Oriented Dialogue](https://mlsanity.com/paper/1705.05414/) (2017)
- [A Knowledge-Grounded Neural Conversation Model](https://mlsanity.com/paper/1702.01932/) (2017)
- [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://mlsanity.com/paper/1908.10763/) (2019)
- [Learning End-to-End Goal-Oriented Dialog](https://mlsanity.com/paper/1605.07683/) (2016)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Learning from others' mistakes: Avoiding dataset biases without modeling them](https://mlsanity.com/paper/2012.01300/) (2020)
- [Two Causal Principles for Improving Visual Dialog](https://mlsanity.com/paper/1911.10496/) (2019)

## Built on by (most cited first, 4 of 4)

- [Efficient Task-Oriented Dialogue Systems with Response Selection as an Auxiliary Task](https://mlsanity.com/paper/2208.07097/) (2022)
- [Multijugate Dual Learning for Low-Resource Task-Oriented Dialogue System](https://mlsanity.com/paper/2305.16106/) (2023)
- [Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems](https://mlsanity.com/paper/2307.07255/) (2023)
- [Spec-TOD: A Specialized Instruction-Tuned LLM Framework for Efficient Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2507.04841/) (2025)

## Abstract (arXiv)

Data artifacts incentivize machine learning models to learn non-transferable generalizations by taking advantage of shortcuts in the data, and there is growing evidence that data artifacts play a role for the strong results that deep learning models achieve in recent natural language processing benchmarks. In this paper, we focus on task-oriented dialogue and investigate whether popular datasets such as MultiWOZ contain such data artifacts. We found that by only keeping frequent phrases in the training examples, state-of-the-art models perform similarly compared to the variant trained with full data, suggesting they exploit these spurious correlations to solve the task. Motivated by this, we propose a contrastive learning based framework to encourage the model to ignore these cues and focus on learning generalisable patterns. We also experiment with adversarial filtering to remove "easy" training instances so that the model would focus on learning from the "harder" instances. We conduct a number of generalization experiments -- e.g., cross-domain/dataset and adversarial tests -- to assess the robustness of our approach and found that it works exceptionally well.