# Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis

arXiv: https://arxiv.org/abs/2107.02568 · HTML: https://mlsanity.com/paper/2107.02568/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study](https://mlsanity.com/paper/2004.03271/) (2020)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions](https://mlsanity.com/paper/2104.03829/) (2021)
- [Unsupervised Lesion Detection via Image Restoration with a Normative Prior](https://mlsanity.com/paper/2005.00031/) (2020)
- [Detecting Outliers with Foreign Patch Interpolation](https://mlsanity.com/paper/2011.04197/) (2020)
- [Unsupervised Brain Anomaly Detection and Segmentation with Transformers](https://mlsanity.com/paper/2102.11650/) (2021)
- [Bayesian Image Quality Transfer with CNNs: Exploring Uncertainty in dMRI Super-Resolution](https://mlsanity.com/paper/1705.00664/) (2017)

## Built on by (most cited first, 30 of 34)

- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [Test-Time Model Adaptation with Only Forward Passes](https://mlsanity.com/paper/2404.01650/) (2024)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [A framework for benchmarking class-out-of-distribution detection and its application to ImageNet](https://mlsanity.com/paper/2302.11893/) (2023)
- [Out-of-distribution Detection in Medical Image Analysis: A survey](https://mlsanity.com/paper/2404.18279/) (2024)
- [Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2403.11491/) (2024)
- [CertainNet: Sampling-free Uncertainty Estimation for Object Detection](https://mlsanity.com/paper/2110.01604/) (2021)
- [On the use of Mahalanobis distance for out-of-distribution detection with neural networks for medical imaging](https://mlsanity.com/paper/2309.01488/) (2023)
- [Towards out of distribution generalization for problems in mechanics](https://mlsanity.com/paper/2206.14917/) (2022)
- [Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?](https://mlsanity.com/paper/2505.20903/) (2025)
- [Recommendations on test datasets for evaluating AI solutions in pathology](https://mlsanity.com/paper/2204.14226/) (2022)
- [Surgical Workflow Recognition: from Analysis of Challenges to Architectural Study](https://mlsanity.com/paper/2203.09230/) (2022)
- [Dimensionality Reduction for Improving Out-of-Distribution Detection in Medical Image Segmentation](https://mlsanity.com/paper/2308.03723/) (2023)
- [Test for non-negligible adverse shifts](https://mlsanity.com/paper/2107.02990/) (2021)
- [Is it all a cluster game? -- Exploring Out-of-Distribution Detection based on Clustering in the Embedding Space](https://mlsanity.com/paper/2203.08549/) (2022)
- [Improving Uncertainty-based Out-of-Distribution Detection for Medical Image Segmentation](https://mlsanity.com/paper/2211.05421/) (2022)
- [Multi-layer Aggregation as a key to feature-based OOD detection](https://mlsanity.com/paper/2307.15647/) (2023)
- [NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds](https://mlsanity.com/paper/2206.11736/) (2022)
- [Test Time Transform Prediction for Open Set Histopathological Image Recognition](https://mlsanity.com/paper/2206.10033/) (2022)
- [nnOOD: A Framework for Benchmarking Self-supervised Anomaly Localisation Methods](https://mlsanity.com/paper/2209.01124/) (2022)
- [Distance Matters For Improving Performance Estimation Under Covariate Shift](https://mlsanity.com/paper/2308.07223/) (2023)
- [Situation Monitor: Diversity-Driven Zero-Shot Out-of-Distribution Detection using Budding Ensemble Architecture for Object Detection](https://mlsanity.com/paper/2406.03188/) (2024)
- [Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impact](https://mlsanity.com/paper/2412.19124/) (2024)
- [ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization](https://mlsanity.com/paper/2603.14254/) (2026)
- [Which models are innately best at uncertainty estimation?](https://mlsanity.com/paper/2206.02152/) (2022)
- [Uncertainty estimations methods for a deep learning model to aid in clinical decision-making -- a clinician's perspective](https://mlsanity.com/paper/2210.00589/) (2022)
- [Unsupervised out-of-distribution detection for safer robotically guided retinal microsurgery](https://mlsanity.com/paper/2304.05040/) (2023)
- [DAGnosis: Localized Identification of Data Inconsistencies using Structures](https://mlsanity.com/paper/2402.17599/) (2024)
- [Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentation](https://mlsanity.com/paper/2512.08216/) (2025)
- [Exploring How Anomalous Model Input and Output Alerts Affect Decision-Making in Healthcare](https://mlsanity.com/paper/2204.13194/) (2022)

## Abstract (arXiv)

Image classification models deployed in the real world may receive inputs outside the intended data distribution. For critical applications such as clinical decision making, it is important that a model can detect such out-of-distribution (OOD) inputs and express its uncertainty. In this work, we assess the capability of various state-of-the-art approaches for confidence-based OOD detection through a comparative study and in-depth analysis. First, we leverage a computer vision benchmark to reproduce and compare multiple OOD detection methods. We then evaluate their capabilities on the challenging task of disease classification using chest X-rays. Our study shows that high performance in a computer vision task does not directly translate to accuracy in a medical imaging task. We analyse factors that affect performance of the methods between the two tasks. Our results provide useful insights for developing the next generation of OOD detection methods.