# Improving Robustness of Task Oriented Dialog Systems

arXiv: https://arxiv.org/abs/1911.05153 · HTML: https://mlsanity.com/paper/1911.05153/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2019.

## Built on by (most cited first, 15 of 15)

- [SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems](https://mlsanity.com/paper/2110.06800/) (2021)
- [AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models](https://mlsanity.com/paper/2102.05126/) (2021)
- [Semantic Representation for Dialogue Modeling](https://mlsanity.com/paper/2105.10188/) (2021)
- [TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling](https://mlsanity.com/paper/2109.04562/) (2021)
- [Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding](https://mlsanity.com/paper/2104.06393/) (2021)
- [On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise](https://mlsanity.com/paper/2104.07149/) (2021)
- [Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining](https://mlsanity.com/paper/2210.04782/) (2022)
- [Neural model robustness for skill routing in large-scale conversational AI systems: A design choice exploration](https://mlsanity.com/paper/2103.03373/) (2021)
- [Multi-Granularity Prompts for Topic Shift Detection in Dialogue](https://mlsanity.com/paper/2305.14006/) (2023)
- [Topic Shift Detection in Chinese Dialogues: Corpus and Benchmark](https://mlsanity.com/paper/2305.01195/) (2023)
- [Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties](https://mlsanity.com/paper/2402.02078/) (2024)
- [Learning towards Selective Data Augmentation for Dialogue Generation](https://mlsanity.com/paper/2303.09719/) (2023)
- [More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking](https://mlsanity.com/paper/2303.09905/) (2023)
- [Data Augmentation for Improving Tail-traffic Robustness in Skill-routing for Dialogue Systems](https://mlsanity.com/paper/2306.04823/) (2023)
- [Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variations](https://mlsanity.com/paper/2311.08705/) (2023)

## Abstract (arXiv)

Task oriented language understanding in dialog systems is often modeled using intents (task of a query) and slots (parameters for that task). Intent detection and slot tagging are, in turn, modeled using sentence classification and word tagging techniques respectively. Similar to adversarial attack problems with computer vision models discussed in existing literature, these intent-slot tagging models are often over-sensitive to small variations in input -- predicting different and often incorrect labels when small changes are made to a query, thus reducing their accuracy and reliability. However, evaluating a model's robustness to these changes is harder for language since words are discrete and an automated change (e.g. adding `noise') to a query sometimes changes the meaning and thus labels of a query. In this paper, we first describe how to create an adversarial test set to measure the robustness of these models. Furthermore, we introduce and adapt adversarial training methods as well as data augmentation using back-translation to mitigate these issues. Our experiments show that both techniques improve the robustness of the system substantially and can be combined to yield the best results.