# No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets

arXiv: https://arxiv.org/abs/2109.05554 · HTML: https://mlsanity.com/paper/2109.05554/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 12 Sep 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](https://mlsanity.com/paper/2105.01879/) (2021)
- [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://mlsanity.com/paper/2106.09022/) (2021)
- [Detecting semantic anomalies](https://mlsanity.com/paper/1908.04388/) (2019)
- [SelfAugment: Automatic Augmentation Policies for Self-Supervised Learning](https://mlsanity.com/paper/2009.07724/) (2020)
- [CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting](https://mlsanity.com/paper/2002.11379/) (2020)

## Built on by (most cited first, 30 of 31)

- [OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection](https://mlsanity.com/paper/2306.09301/) (2023)
- [A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks](https://mlsanity.com/paper/2112.00337/) (2021)
- [Out-of-Distribution Detection Methods Answer the Wrong Questions](https://mlsanity.com/paper/2507.01831/) (2025)
- [WDiscOOD: Out-of-Distribution Detection via Whitened Linear Discriminant Analysis](https://mlsanity.com/paper/2303.07543/) (2023)
- [Conservative Prediction via Data-Driven Confidence Minimization](https://mlsanity.com/paper/2306.04974/) (2023)
- [Bridging Streaming Continual Learning via In-Context Large Tabular Models](https://mlsanity.com/paper/2512.11668/) (2025)
- [Out of Distribution Detection on ImageNet-O](https://mlsanity.com/paper/2201.09352/) (2022)
- [COOD: Combined out-of-distribution detection using multiple measures for anomaly & novel class detection in large-scale hierarchical classification](https://mlsanity.com/paper/2403.06874/) (2024)
- [WeiPer: OOD Detection using Weight Perturbations of Class Projections](https://mlsanity.com/paper/2405.17164/) (2024)
- [Improving Out-of-Distribution Detection via Dynamic Covariance Calibration](https://mlsanity.com/paper/2506.09399/) (2025)
- [BOOD: Boundary-based Out-Of-Distribution Data Generation](https://mlsanity.com/paper/2508.00350/) (2025)
- [Building Real-time Awareness of Out-of-distribution in Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2409.17277/) (2024)
- [A Recipe for Causal Graph Regression: Confounding Effects Revisited](https://mlsanity.com/paper/2507.00440/) (2025)
- [Interpretable Out-Of-Distribution Detection Using Pattern Identification](https://mlsanity.com/paper/2302.10303/) (2023)
- [Monitizer: Automating Design and Evaluation of Neural Network Monitors](https://mlsanity.com/paper/2405.10350/) (2024)
- [STOOD-X methodology: using statistical nonparametric test for OOD Detection Large-Scale datasets enhanced with explainability](https://mlsanity.com/paper/2504.02685/) (2025)
- [GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detection](https://mlsanity.com/paper/2505.16017/) (2025)
- [COMBOOD: A Semiparametric Approach for Detecting Out-of-distribution Data for Image Classification](https://mlsanity.com/paper/2602.07042/) (2026)
- [XOOD: Extreme Value Based Out-Of-Distribution Detection For Image Classification](https://mlsanity.com/paper/2208.00629/) (2022)
- [Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Models](https://mlsanity.com/paper/2211.15718/) (2022)
- [A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifiers](https://mlsanity.com/paper/2308.04137/) (2023)
- [Toward a Realistic Benchmark for Out-of-Distribution Detection](https://mlsanity.com/paper/2404.10474/) (2024)
- [TASTE: Task-Aware Out-of-Distribution Detection via Stein Operators](https://mlsanity.com/paper/2602.07640/) (2026)
- [Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning](https://mlsanity.com/paper/2605.05638/) (2026)
- [Simple Calibration via Geodesic Kernels](https://mlsanity.com/paper/2201.13001/) (2022)
- [Out of Distribution Detection for Efficient Continual Learning in Quality Prediction for Arc Welding](https://mlsanity.com/paper/2508.16832/) (2025)
- [Benchmarking Out-of-Distribution Detection for Plankton Recognition: A Systematic Evaluation of Advanced Methods in Marine Ecological Monitoring](https://mlsanity.com/paper/2510.17179/) (2025)
- [The Invisible Gorilla Effect in Out-of-distribution Detection](https://mlsanity.com/paper/2602.20068/) (2026)
- [Bounding Box Anomaly Scoring for simple and efficient Out-of-Distribution detection](https://mlsanity.com/paper/2603.22660/) (2026)
- [Few-Shot Out of Domain Intent Detection with Covariance Corrected Mahalanobis Distance](https://mlsanity.com/paper/2609.00961/) (2026)

## Abstract (arXiv)

Out-of-distribution detection is an important component of reliable ML systems. Prior literature has proposed various methods (e.g., MSP (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018)), claiming they are state-of-the-art by showing they outperform previous methods on a selected set of in-distribution (ID) and out-of-distribution (OOD) datasets. In this work, we show that none of these methods are inherently better at OOD detection than others on a standardized set of 16 (ID, OOD) pairs. We give possible explanations for these inconsistencies with simple toy datasets where whether one method outperforms another depends on the structure of the ID and OOD datasets in question. Finally, we show that a method outperforming another on a certain (ID, OOD) pair may not do so in a low-data regime. In the low-data regime, we propose a distance-based method, Pairwise OOD detection (POD), which is based on Siamese networks and improves over Mahalanobis by sidestepping the expensive covariance estimation step. Our results suggest that the OOD detection problem may be too broad, and we should consider more specific structures for leverage.