mlsanityData to 7 Oct 2026

Types of Out-of-Distribution Texts and How to Detect Them

arXiv 2109.06827 ↗First posted 14 Sep 2021cs.CL cs.LGcited by 80

Not on the map. Why?

Abstract (arXiv)

Despite agreement on the importance of detecting out-of-distribution (OOD) examples, there is little consensus on the formal definition of OOD examples and how to best detect them. We categorize these examples by whether they exhibit a background shift or a semantic shift, and find that the two major approaches to OOD detection, model calibration and density estimation (language modeling for text), have distinct behavior on these types of OOD data. Across 14 pairs of in-distribution and OOD English natural language understanding datasets, we find that density estimation methods consistently beat calibration methods in background shift settings, while performing worse in semantic shift settings. In addition, we find that both methods generally fail to detect examples from challenge data, highlighting a weak spot for current methods. Since no single method works well across all settings, our results call for an explicit definition of OOD examples when evaluating different detection methods.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

43 of 60 references.

References · 43

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  12. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
31 more
  1. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  5. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  6. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  7. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  8. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  9. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  10. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  11. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  12. 2020Calibration of Pre-trained Transformerscited by 307
  13. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  14. 2020Selective Question Answering under Domain Shiftcited by 228
  15. 2020Why Normalizing Flows Fail to Detect Out-of-Distribution Datacited by 220
  16. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  17. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  18. 2017Style Transfer in Text: Exploration and Evaluationcited by 145
  19. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  20. 2020Hybrid Models for Open Set Recognitioncited by 109
  21. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 70
  22. 2018A Survey of Domain Adaptation for Neural Machine Translationcited by 64
  23. 2019Out-of-domain Detection for Natural Language Understanding in Dialog Systemscited by 62
  24. 2019Detecting semantic anomaliescited by 50
  25. 2019Hybrid Models with Deep and Invertible Featurescited by 41
  26. 2019Out-of-Domain Detection for Low-Resource Text Classification Taskscited by 36
  27. 2017Exposure: A White-Box Photo Post-Processing Frameworkcited by 27
  28. 2020Hybrid Discriminative-Generative Training via Contrastive Learningcited by 24
  29. 2020Domain Divergences: a Survey and Empirical Analysiscited by 24
  30. 2020Misinformation Has High Perplexitycited by 15
  31. 2020Natural Language Processing with Deep Learning for Medical Adverse Event Detection from Free-Text Medical Narratives: A Case Study of Detecting Total Hip Replacement Dislocationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 35; 2024: 11; 2025: 9; 2026: 5.

80 papers.

Sort:

By area:

  1. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  2. 2022Prompting GPT-3 To Be Reliablecited by 258
  3. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  4. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  5. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  6. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  7. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  8. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  9. 2024Safeguarding Large Language Models: A Surveycited by 73
  10. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  11. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  12. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
38 more
  1. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  2. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  3. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  4. 2025Estimating LLM Uncertainty with Evidencecited by 26
  5. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
  6. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 24
  7. 2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
  8. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  9. 2023A Survey on Out-of-Distribution Evaluation of Neural NLP Modelscited by 20
  10. 2023ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessmentcited by 19
  11. 2025Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMscited by 19
  12. 2024Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoningcited by 18
  13. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  14. 2023Unsupervised Layer-wise Score Aggregation for Textual OOD Detectioncited by 15
  15. 2023How Good Are LLMs at Out-of-Distribution Detection?cited by 15
  16. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 15
  17. 2022Measures of Information Reflect Memorization Patternscited by 12
  18. 2023Mask-then-Fill: A Flexible and Effective Data Augmentation Framework for Event Extractioncited by 12
  19. 2023Fine-Tuning Deteriorates General Textual Out-of-Distribution Detection by Distorting Task-Agnostic Featurescited by 12
  20. 2022Active Learning Over Multiple Domains in Natural Language Taskscited by 11
  21. 2022Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Textcited by 11
  22. 2023Few-shot Anomaly Detection in Text with Deviation Learningcited by 11
  23. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
  24. 2022Benchmarking Long-tail Generalization with Likelihood Splitscited by 8
  25. 2023Reliable Gradient-free and Likelihood-free Prompt Tuningcited by 8
  26. 2022Holistic Sentence Embeddings for Better Out-of-Distribution Detectioncited by 6
  27. 2023Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editorscited by 6
  28. 2024Generalizing Trust: Weak-to-Strong Trustworthiness in Language Modelscited by 6
  29. 2025Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detectioncited by 6
  30. 2022Self-Supervised Losses for One-Class Textual Anomaly Detectioncited by 5
  31. 2022SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Modelscited by 5
  32. 2023DeepLens: Interactive Out-of-distribution Data Detection in NLP Modelscited by 5
  33. 2023Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Taskscited by 5
  34. 2023Pseudo Outlier Exposure for Out-of-Distribution Detection using Pretrained Transformerscited by 5
  35. 2023Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answeringcited by 5
  36. 2023FLatS: Principled Out-of-Distribution Detection with Feature-Based Likelihood Ratio Scorecited by 5
  37. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 5
  38. 2023MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognitioncited by 4

50 most cited of 80 All 80 (JSON)