# Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox

arXiv: https://arxiv.org/abs/2406.09867 · HTML: https://mlsanity.com/paper/2406.09867/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [Extremely Simple Activation Shaping for Out-of-Distribution Detection](https://mlsanity.com/paper/2209.09858/) (2022)
- [OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection](https://mlsanity.com/paper/2306.09301/) (2023)
- [DICE: Leveraging Sparsification for Out-of-Distribution Detection](https://mlsanity.com/paper/2111.09805/) (2021)
- [In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation](https://mlsanity.com/paper/2306.00826/) (2023)
- [Semantically Coherent Out-of-Distribution Detection](https://mlsanity.com/paper/2108.11941/) (2021)
- [Full-Spectrum Out-of-Distribution Detection](https://mlsanity.com/paper/2204.05306/) (2022)
- [RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection](https://mlsanity.com/paper/2209.08590/) (2022)
- [A framework for benchmarking class-out-of-distribution detection and its application to ImageNet](https://mlsanity.com/paper/2302.11893/) (2023)
- [ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms](https://mlsanity.com/paper/2310.01755/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models](https://mlsanity.com/paper/2501.18463/) (2025)
- [DCV-ROOD Evaluation Framework: Dual Cross-Validation for Robust Out-of-Distribution Detection](https://mlsanity.com/paper/2509.05778/) (2025)
- [Out-of-Distribution Detection from Small Training Sets using Bayesian Neural Network Classifiers](https://mlsanity.com/paper/2510.06025/) (2025)

## Abstract (arXiv)

Most existing out-of-distribution (OOD) detection benchmarks classify samples with novel labels as the OOD data. However, some marginal OOD samples actually have close semantic contents to the in-distribution (ID) sample, which makes determining the OOD sample a Sorites Paradox. In this paper, we construct a benchmark named Incremental Shift OOD (IS-OOD) to address the issue, in which we divide the test samples into subsets with different semantic and covariate shift degrees relative to the ID dataset. The data division is achieved through a shift measuring method based on our proposed Language Aligned Image feature Decomposition (LAID). Moreover, we construct a Synthetic Incremental Shift (Syn-IS) dataset that contains high-quality generated images with more diverse covariate contents to complement the IS-OOD benchmark. We evaluate current OOD detection methods on our benchmark and find several important insights: (1) The performance of most OOD detection methods significantly improves as the semantic shift increases; (2) Some methods like GradNorm may have different OOD detection mechanisms as they rely less on semantic shifts to make decisions; (3) Excessive covariate shifts in the image are also likely to be considered as OOD for some methods. Our code and data are released in https://github.com/qqwsad5/IS-OOD.