mlsanityData to 7 Oct 2026

In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation

arXiv 2306.00826 ↗First posted 1 Jun 2023cs.LG cs.CVcited by 132

Not on the map. Why?

Abstract (arXiv)

Out-of-distribution (OOD) detection is the problem of identifying inputs which are unrelated to the in-distribution task. The OOD detection performance when the in-distribution (ID) is ImageNet-1K is commonly being tested on a small range of test OOD datasets. We find that most of the currently used test OOD datasets, including datasets from the open set recognition (OSR) literature, have severe issues: In some cases more than 50% of the dataset contains objects belonging to one of the ID classes. These erroneous samples heavily distort the evaluation of OOD detectors. As a solution, we introduce with NINCO a novel test OOD dataset, each sample checked to be ID free, which with its fine-grained range of OOD classes allows for a detailed analysis of an OOD detector's strengths and failure modes, particularly when paired with a number of synthetic "OOD unit-tests". We provide detailed evaluations across a large set of architectures and OOD detection methods on NINCO and the unit-tests, revealing new insights about model weaknesses and the effects of pretraining on OOD detection performance. We provide code and data at https://github.com/j-cb/NINCO.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

36 of 51 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  9. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  10. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  12. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
24 more
  1. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  2. 2015Towards Open Set Deep Networkscited by 533
  3. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  4. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  5. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  6. 2022DeiT III: Revenge of the ViTcited by 417
  7. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
  8. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  9. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  10. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  11. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  12. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  13. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  14. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 206
  15. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  16. 2021Can multi-label classification networks know what they don't know?cited by 100
  17. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 60
  18. 2022RankFeat: Rank-1 Feature Removal for Out-of-distribution Detectioncited by 52
  19. 2022Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognitioncited by 44
  20. 2023A framework for benchmarking class-out-of-distribution detection and its application to ImageNetcited by 31
  21. 2021OODformer: Out-Of-Distribution Detection Transformercited by 27
  22. 2022Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantitiescited by 23
  23. 2021Provably Robust Detection of Out-of-distribution Data (almost) for freecited by 17
  24. 2022On the Usefulness of Deep Ensemble Diversity for Out-of-Distribution Detectioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 40; 2025: 42; 2026: 34.

132 papers.

Sort:

By area:

  1. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  2. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  3. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  4. 2023Fast Decision Boundary based Out-of-Distribution Detectorcited by 48
  5. 2023NECO: NEural Collapse Based Out-of-distribution detectioncited by 41
  6. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  7. 2024Outlier detection by ensembling uncertainty with negative objectnesscited by 34
  8. 2023Detecting Out-of-Distribution Through the Lens of Neural Collapsecited by 33
  9. 2023Don't trust your eyes: on the (un)reliability of feature visualizationscited by 32
  10. 2024COME: Test-time adaption by Conservatively Minimizing Entropycited by 31
  11. 2024Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detectioncited by 30
  12. 2025Mahalanobis++: Improving OOD Detection via Feature Normalizationcited by 30
38 more
  1. 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
  2. 2024The Entropy Enigma: Success and Failure of Entropy Minimizationcited by 24
  3. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 24
  4. 2023ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithmscited by 22
  5. 2024Out-of-Distribution Detection with a Single Unconditional Diffusion Modelcited by 22
  6. 2024Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domainscited by 21
  7. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Modelscited by 21
  8. 2025Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learningcited by 21
  9. 2023From Global to Local: Multi-scale Out-of-distribution Detectioncited by 18
  10. 2024MetaOOD: Automatic Selection of OOD Detection Modelscited by 16
  11. 2024AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Modelscited by 16
  12. 2025OODD: Test-time Out-of-Distribution Detection with Dynamic Dictionarycited by 16
  13. 2025Reliable Few-shot Learning under Dual Noisescited by 16
  14. 2025Out-of-Distribution Detection Methods Answer the Wrong Questionscited by 14
  15. 2024A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?cited by 13
  16. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  17. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  18. 2025OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detectioncited by 13
  19. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  20. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  21. 2025ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoningcited by 10
  22. 2025Probing the Representational Power of Sparse Autoencoders in Vision Modelscited by 9
  23. 2024Towards flexible perception with visual memorycited by 8
  24. 2025Noisy Test-Time Adaptation in Vision-Language Modelscited by 8
  25. 2025CADRef: Robust Out-of-Distribution Detection via Class-Aware Decoupled Relative Feature Leveragingcited by 8
  26. 2024A Unified Approach Towards Active Learning and Out-of-Distribution Detectioncited by 7
  27. 2024Out-Of-Distribution Detection with Diversification (Provably)cited by 6
  28. 2023Classifier-head Informed Feature Masking and Prototype-based Logit Smoothing for Out-of-Distribution Detectioncited by 5
  29. 2023Likelihood-Aware Semantic Alignment for Full-Spectrum Out-of-Distribution Detectioncited by 5
  30. 2024COOD: Combined out-of-distribution detection using multiple measures for anomaly & novel class detection in large-scale hierarchical classificationcited by 5
  31. 2025Leveraging Intermediate Representations for Better Out-of-Distribution Detectioncited by 5
  32. 2025Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detectioncited by 5
  33. 2025Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformerscited by 5
  34. 2024Your Classifier Can Be Secretly a Likelihood-Based OOD Detectorcited by 4
  35. 2024VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detectioncited by 4
  36. 2024What If the Input is Expanded in OOD Detection?cited by 4
  37. 2024Out-of-Distribution Detection with Prototypical Outlier Proxycited by 4
  38. 2025BOOD: Boundary-based Out-Of-Distribution Data Generationcited by 4

50 most cited of 132 All 132 (JSON)