# On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise

arXiv: https://arxiv.org/abs/2104.07149 · HTML: https://mlsanity.com/paper/2104.07149/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Combating Adversarial Misspellings with Robust Word Recognition](https://mlsanity.com/paper/1905.11268/) (2019)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://mlsanity.com/paper/1906.02443/) (2019)
- [It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations](https://mlsanity.com/paper/2005.04364/) (2020)
- [Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation](https://mlsanity.com/paper/1902.01509/) (2019)
- [Evaluating Robustness to Input Perturbations for Neural Machine Translation](https://mlsanity.com/paper/2005.00580/) (2020)
- [Misspelling Oblivious Word Embeddings](https://mlsanity.com/paper/1905.09755/) (2019)
- [Improving Robustness of Task Oriented Dialog Systems](https://mlsanity.com/paper/1911.05153/) (2019)

## Built on by (most cited first, 9 of 9)

- [A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialog](https://mlsanity.com/paper/2207.13211/) (2022)
- [Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining](https://mlsanity.com/paper/2210.04782/) (2022)
- [Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors](https://mlsanity.com/paper/2310.16609/) (2023)
- [Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties](https://mlsanity.com/paper/2402.02078/) (2024)
- [Domain Adaptation in Intent Classification Systems: A Review](https://mlsanity.com/paper/2404.14415/) (2024)
- [Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning](https://mlsanity.com/paper/2401.04361/) (2024)
- [Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders](https://mlsanity.com/paper/2403.04314/) (2024)
- [Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification](https://mlsanity.com/paper/2403.05640/) (2024)
- [Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variations](https://mlsanity.com/paper/2311.08705/) (2023)

## Abstract (arXiv)

Intent Classification (IC) and Slot Labeling (SL) models, which form the basis of dialogue systems, often encounter noisy data in real-word environments. In this work, we investigate how robust IC/SL models are to noisy data. We collect and publicly release a test-suite for seven common noise types found in production human-to-bot conversations (abbreviations, casing, misspellings, morphological variants, paraphrases, punctuation and synonyms). On this test-suite, we show that common noise types substantially degrade the IC accuracy and SL F1 performance of state-of-the-art BERT-based IC/SL models. By leveraging cross-noise robustness transfer -- training on one noise type to improve robustness on another noise type -- we design aggregate data-augmentation approaches that increase the model performance across all seven noise types by +10.8% for IC accuracy and +15 points for SL F1 on average. To the best of our knowledge, this is the first work to present a single IC/SL model that is robust to a wide range of noise phenomena.