# A Benchmark of Medical Out of Distribution Detection

arXiv: https://arxiv.org/abs/2007.04250 · HTML: https://mlsanity.com/paper/2007.04250/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2020.

## Built on by (most cited first, 30 of 40)

- [Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild](https://mlsanity.com/paper/2203.03800/) (2022)
- [Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions](https://mlsanity.com/paper/2104.03829/) (2021)
- [Learning to Augment Distributions for Out-of-Distribution Detection](https://mlsanity.com/paper/2311.01796/) (2023)
- [How is BERT surprised? Layerwise detection of linguistic anomalies](https://mlsanity.com/paper/2105.07452/) (2021)
- [Out-of-distribution Detection in Medical Image Analysis: A survey](https://mlsanity.com/paper/2404.18279/) (2024)
- [Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection](https://mlsanity.com/paper/2110.14019/) (2021)
- [GOLD: Graph Out-of-Distribution Detection via Implicit Adversarial Latent Generation](https://mlsanity.com/paper/2502.05780/) (2025)
- [OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detection](https://mlsanity.com/paper/2503.16247/) (2025)
- [Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection](https://mlsanity.com/paper/2405.17816/) (2024)
- [Recommendations on test datasets for evaluating AI solutions in pathology](https://mlsanity.com/paper/2204.14226/) (2022)
- [Are We Ready for Out-of-Distribution Detection in Digital Pathology?](https://mlsanity.com/paper/2407.13708/) (2024)
- [CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI](https://mlsanity.com/paper/2202.02833/) (2022)
- [Out-of-distribution detection algorithms for robust insect classification](https://mlsanity.com/paper/2305.01823/) (2023)
- [A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1](https://mlsanity.com/paper/2312.17506/) (2023)
- [CVAD: A generic medical anomaly detector based on Cascade VAE](https://mlsanity.com/paper/2110.15811/) (2021)
- [Dimensionality Reduction for Improving Out-of-Distribution Detection in Medical Image Segmentation](https://mlsanity.com/paper/2308.03723/) (2023)
- [An Uncertainty-Informed Framework for Trustworthy Fault Diagnosis in Safety-Critical Applications](https://mlsanity.com/paper/2111.00874/) (2021)
- [Confidence-Aware and Self-Supervised Image Anomaly Localisation](https://mlsanity.com/paper/2303.13227/) (2023)
- [CheXseen: Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays](https://mlsanity.com/paper/2103.04590/) (2021)
- [Out-of-Distribution Detection in Time-Series Domain: A Novel Seasonal Ratio Scoring Approach](https://mlsanity.com/paper/2207.04306/) (2022)
- [Key Feature Replacement of In-Distribution Samples for Out-of-Distribution Detection](https://mlsanity.com/paper/2301.13012/) (2022)
- [Redesigning Out-of-Distribution Detection on 3D Medical Images](https://mlsanity.com/paper/2308.07324/) (2023)
- [Margin-Aware Intra-Class Novelty Identification for Medical Images](https://mlsanity.com/paper/2108.00117/) (2021)
- [MLHOps: Machine Learning for Healthcare Operations](https://mlsanity.com/paper/2305.02474/) (2023)
- [Uncertainty Estimation and Out-of-Distribution Detection for Deep Learning-Based Image Reconstruction using the Local Lipschitz](https://mlsanity.com/paper/2305.07618/) (2023)
- [Machine Learning Applications on Neuroimaging for Diagnosis and Prognosis of Epilepsy: A Review](https://mlsanity.com/paper/2102.03336/) (2021)
- [Thinkback: Task-SpecificOut-of-Distribution Detection](https://mlsanity.com/paper/2107.06668/) (2021)
- [A Novel Explainable Out-of-Distribution Detection Approach for Spiking Neural Networks](https://mlsanity.com/paper/2210.00894/) (2022)
- [DIAGNOSE: Avoiding Out-of-distribution Data using Submodular Information Measures](https://mlsanity.com/paper/2210.01526/) (2022)
- [Out-of-distribution Reject Option Method for Dataset Shift Problem in Early Disease Onset Prediction](https://mlsanity.com/paper/2405.19864/) (2024)

## Abstract (arXiv)

Motivation: Deep learning models deployed for use on medical tasks can be equipped with Out-of-Distribution Detection (OoDD) methods in order to avoid erroneous predictions. However it is unclear which OoDD method should be used in practice. Specific Problem: Systems trained for one particular domain of images cannot be expected to perform accurately on images of a different domain. These images should be flagged by an OoDD method prior to diagnosis. Our approach: This paper defines 3 categories of OoD examples and benchmarks popular OoDD methods in three domains of medical imaging: chest X-ray, fundus imaging, and histology slides. Results: Our experiments show that despite methods yielding good results on some categories of out-of-distribution samples, they fail to recognize images close to the training distribution. Conclusion: We find a simple binary classifier on the feature representation has the best accuracy and AUPRC on average. Users of diagnostic tools which employ these OoDD methods should still remain vigilant that images very close to the training distribution yet not in it could yield unexpected results.