# Outlier Detection for Improved Data Quality and Diversity in Dialog Systems

arXiv: https://arxiv.org/abs/1904.03122 · HTML: https://mlsanity.com/paper/1904.03122/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 28 of 28)

- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness](https://mlsanity.com/paper/2305.12947/) (2023)
- [Semantic Diversity in Dialogue with Natural Language Inference](https://mlsanity.com/paper/2205.01497/) (2022)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation](https://mlsanity.com/paper/2401.06643/) (2024)
- [A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialog](https://mlsanity.com/paper/2207.13211/) (2022)
- [Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Health](https://mlsanity.com/paper/2109.13770/) (2021)
- [Towards Robustness to Label Noise in Text Classification via Noise Modeling](https://mlsanity.com/paper/2101.11214/) (2021)
- [ActiveAED: A Human in the Loop Improves Annotation Error Detection](https://mlsanity.com/paper/2305.20045/) (2023)
- [NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2204.13021/) (2022)
- [Redwood: Using Collision Detection to Grow a Large-Scale Intent Classification Dataset](https://mlsanity.com/paper/2204.05483/) (2022)
- [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023)
- [TempoFormer: A Transformer for Temporally-aware Representations in Change Detection](https://mlsanity.com/paper/2408.15689/) (2024)
- [Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets](https://mlsanity.com/paper/2205.11472/) (2022)
- [A survey of diversity quantification in natural language processing: The why, what, where and how](https://mlsanity.com/paper/2507.20858/) (2025)
- [Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning](https://mlsanity.com/paper/2111.13461/) (2021)
- [Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification](https://mlsanity.com/paper/2403.05640/) (2024)
- [Crowdsourcing Diverse Paraphrases for Training Task-oriented Bots](https://mlsanity.com/paper/2109.09420/) (2021)
- [Anomaly-Injected Deep Support Vector Data Description for Text Outlier Detection](https://mlsanity.com/paper/2110.14729/) (2021)
- [Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets?](https://mlsanity.com/paper/2309.01669/) (2023)
- [ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMs](https://mlsanity.com/paper/2505.15443/) (2025)
- [Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow](https://mlsanity.com/paper/2302.04434/) (2023)
- [ShabbyPages: A Reproducible Document Denoising and Binarization Dataset](https://mlsanity.com/paper/2303.09339/) (2023)
- [LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity](https://mlsanity.com/paper/2304.06184/) (2023)
- [Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models](https://mlsanity.com/paper/2502.08669/) (2025)
- [EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI](https://mlsanity.com/paper/2511.08949/) (2025)
- [Reliability-Aware LLM Alignment from Inconsistent Human Feedback](https://mlsanity.com/paper/2607.20515/) (2026)

## Abstract (arXiv)

In a corpus of data, outliers are either errors: mistakes in the data that are counterproductive, or are unique: informative samples that improve model robustness. Identifying outliers can lead to better datasets by (1) removing noise in datasets and (2) guiding collection of additional data to fill gaps. However, the problem of detecting both outlier types has received relatively little attention in NLP, particularly for dialog systems. We introduce a simple and effective technique for detecting both erroneous and unique samples in a corpus of short texts using neural sentence embeddings combined with distance-based outlier detection. We also present a novel data collection pipeline built atop our detection technique to automatically and iteratively mine unique data samples while discarding erroneous samples. Experiments show that our outlier detection technique is effective at finding errors while our data collection pipeline yields highly diverse corpora that in turn produce more robust intent classification and slot-filling models.