# Robustness Testing of Language Understanding in Task-Oriented Dialog

arXiv: https://arxiv.org/abs/2012.15262 · HTML: https://mlsanity.com/paper/2012.15262/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 30 of 32)

- [SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems](https://mlsanity.com/paper/2110.06800/) (2021)
- [DS-TOD: Efficient Domain Specialization for Task Oriented Dialog](https://mlsanity.com/paper/2110.08395/) (2021)
- [Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering](https://mlsanity.com/paper/2106.04016/) (2021)
- [Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems](https://mlsanity.com/paper/2112.05842/) (2021)
- [A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialog](https://mlsanity.com/paper/2207.13211/) (2022)
- [TOD-DA: Towards Boosting the Robustness of Task-oriented Dialogue Modeling on Spoken Conversations](https://mlsanity.com/paper/2112.12441/) (2021)
- [Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics](https://mlsanity.com/paper/2112.08321/) (2021)
- [PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling](https://mlsanity.com/paper/2208.11508/) (2022)
- [Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study](https://mlsanity.com/paper/2302.04618/) (2023)
- [Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks](https://mlsanity.com/paper/2502.13645/) (2025)
- [Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts](https://mlsanity.com/paper/2305.03237/) (2023)
- [PromptAttack: Probing Dialogue State Trackers with Adversarial Prompts](https://mlsanity.com/paper/2306.04535/) (2023)
- [Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors](https://mlsanity.com/paper/2310.16609/) (2023)
- [How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task](https://mlsanity.com/paper/2111.09612/) (2021)
- [OLISIA: a Cascade System for Spoken Dialogue State Tracking](https://mlsanity.com/paper/2304.11073/) (2023)
- [Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations](https://mlsanity.com/paper/2310.03518/) (2023)
- [DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Task](https://mlsanity.com/paper/2310.10169/) (2023)
- [Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering](https://mlsanity.com/paper/2205.10363/) (2022)
- [DialogVCS: Robust Natural Language Understanding in Dialogue System Upgrade](https://mlsanity.com/paper/2305.14751/) (2023)
- [Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties](https://mlsanity.com/paper/2402.02078/) (2024)
- [On Sensitivity of Deep Learning Based Text Classification Algorithms to Practical Input Perturbations](https://mlsanity.com/paper/2201.00318/) (2022)
- [Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning](https://mlsanity.com/paper/2209.07873/) (2022)
- [More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking](https://mlsanity.com/paper/2303.09905/) (2023)
- [Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification](https://mlsanity.com/paper/2403.05640/) (2024)
- [Towards Explanation for Unsupervised Graph-Level Representation Learning](https://mlsanity.com/paper/2205.09934/) (2022)
- [BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems](https://mlsanity.com/paper/2211.11982/) (2022)
- [Sources of Noise in Dialogue and How to Deal with Them](https://mlsanity.com/paper/2212.02745/) (2022)
- [DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue](https://mlsanity.com/paper/2212.08054/) (2022)
- [Team Flow at DRC2022: Pipeline System for Travel Destination Recommendation Task in Spoken Dialogue](https://mlsanity.com/paper/2210.09518/) (2022)
- [Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variations](https://mlsanity.com/paper/2311.08705/) (2023)

## Abstract (arXiv)

Most language understanding models in task-oriented dialog systems are trained on a small amount of annotated training data, and evaluated in a small set from the same distribution. However, these models can lead to system failure or undesirable output when being exposed to natural language perturbation or variation in practice. In this paper, we conduct comprehensive evaluation and analysis with respect to the robustness of natural language understanding models, and introduce three important aspects related to language understanding in real-world dialog systems, namely, language variety, speech characteristics, and noise perturbation. We propose a model-agnostic toolkit LAUG to approximate natural language perturbations for testing the robustness issues in task-oriented dialog. Four data augmentation approaches covering the three aspects are assembled in LAUG, which reveals critical robustness issues in state-of-the-art models. The augmented dataset through LAUG can be used to facilitate future research on the robustness testing of language understanding in task-oriented dialog.