mlsanityData to 7 Oct 2026

GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation

arXiv 2109.03079 ↗First posted 7 Sep 2021cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Practical dialogue systems require robust methods of detecting out-of-scope (OOS) utterances to avoid conversational breakdowns and related failure modes. Directly training a model with labeled OOS examples yields reasonable performance, but obtaining such data is a resource-intensive process. To tackle this limited-data problem, previous methods focus on better modeling the distribution of in-scope (INS) examples. We introduce GOLD as an orthogonal technique that augments existing data to train better OOS detectors operating in low-data regimes. GOLD generates pseudo-labeled candidates using samples from an auxiliary dataset and keeps only the most beneficial candidates for training through a novel filtering mechanism. In experiments across three target benchmarks, the top GOLD model outperforms all existing methods on all key metrics, achieving relative gains of 52.4%, 48.9% and 50.3% against median baseline performance. We also analyze the unique properties of OOS data to identify key factors for optimally applying our proposed method.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 45 of its 70 references that are in the map. First posted on arXiv 7 Sep 2021.

Builds on

45 of 70 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  8. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  9. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  10. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  11. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  12. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
33 more
  1. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  2. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  3. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  4. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  5. 2020Selective Question Answering under Domain Shiftcited by 228
  6. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  7. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  8. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  9. 2019A Question-Entailment Approach to Question Answeringcited by 133
  10. 2018Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifierscited by 126
  11. 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 124
  12. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  13. 2016Data Recombination for Neural Semantic Parsingcited by 111
  14. 2017DOC: Deep Open Classification of Text Documentscited by 109
  15. 2018Conditional BERT Contextual Augmentationcited by 107
  16. 2020SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustnesscited by 94
  17. 2018Sequence-to-Sequence Data Augmentation for Dialogue Language Understandingcited by 67
  18. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 67
  19. 2019Deep Unknown Intent Detection with Margin Losscited by 66
  20. 2019Out-of-domain Detection for Natural Language Understanding in Dialog Systemscited by 62
  21. 2020Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generationcited by 47
  22. 2020STAR: A Schema-Guided Dialog Dataset for Transfer Learningcited by 43
  23. 2019Out-of-Domain Detection for Low-Resource Text Classification Taskscited by 36
  24. 2019Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialogcited by 33
  25. 2017Distance-based Confidence Score for Neural Network Classifierscited by 30
  26. 2019Statistical Analysis of Nearest Neighbor Methods for Anomaly Detectioncited by 29
  27. 2019Posing Fair Generalization Tasks for Natural Language Inferencecited by 22
  28. 2019Automatically Learning Data Augmentation Policies for Dialogue Taskscited by 20
  29. 2018Neural Sentence Embedding using Only In-domain Sentences for Out-of-domain Sentence Detection in Dialog Systemscited by 18
  30. 2018Joint Learning of Domain Classification and Out-of-Domain Detection with Dynamic Class Weighting for Satisficing False Acceptance Ratescited by 17
  31. 2021OodGAN: Generative Adversarial Network for Out-of-Domain Data Generationcited by 15
  32. 2020Improving Dialogue Breakdown Detection with Semi-Supervised Learningcited by 6
  33. 2019Contextual Out-of-Domain Utterance Handling With Counterfeit Data Augmentationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 6; 2024: 2; 2025: 0; 2026: 1.

20 papers.

Sort:

  1. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  2. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  3. 2022POEM: Out-of-Distribution Detection with Posterior Samplingcited by 108
  4. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  5. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  6. 2023Mixture of Soft Prompts for Controllable Data Generationcited by 23
  7. 2022Estimating Soft Labels for Out-of-Domain Intent Detectioncited by 11
  8. 2021DG2: Data Augmentation Through Document Grounded Dialogue Generationcited by 10
  9. 2022Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemblecited by 9
  10. 2023Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contextscited by 6
  11. 2024Dual-Adapter: Training-free Dual Adaptation for Few-shot Out-of-Distribution Detectioncited by 4
  12. 2022Data Augmentation for Intent Classificationcited by 3
8 more
  1. 2022Knowledge-Grounded Conversational Data Augmentation with Generative Conversational Networkscited by 2
  2. 2023Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLPcited by 2
  3. 2022Sources of Noise in Dialogue and How to Deal with Themcited by 1
  4. 2022Metric Learning and Adaptive Boundary for Out-of-Domain Detectioncited by 0
  5. 2022Pseudo-OOD training for robust language modelscited by 0
  6. 2023Data Augmentation for Improving Tail-traffic Robustness in Skill-routing for Dialogue Systemscited by 0
  7. 2024Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systemscited by 0
  8. 2026Few-Shot Out of Domain Intent Detection with Covariance Corrected Mahalanobis Distancecited by 0