# Out-of-Distribution Detection for Medical Applications: Guidelines for Practical Evaluation

arXiv: https://arxiv.org/abs/2109.14885 · HTML: https://mlsanity.com/paper/2109.14885/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 27 of 27)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [Why Normalizing Flows Fail to Detect Out-of-Distribution Data](https://mlsanity.com/paper/2006.08545/) (2020)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Understanding Failures in Out-of-Distribution Detection with Deep Generative Models](https://mlsanity.com/paper/2107.06908/) (2021)
- [Discrete Flows: Invertible Generative Models of Discrete Data](https://mlsanity.com/paper/1905.10347/) (2019)
- [VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data](https://mlsanity.com/paper/2006.11941/) (2020)
- [Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Data](https://mlsanity.com/paper/2011.03274/) (2020)
- [Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection](https://mlsanity.com/paper/2012.05329/) (2020)
- [Improving Explorability in Variational Inference with Annealed Variational Objectives](https://mlsanity.com/paper/1809.01818/) (2018)
- [Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimation](https://mlsanity.com/paper/2007.08128/) (2020)
- [Tutorial: Safe and Reliable Machine Learning](https://mlsanity.com/paper/1904.07204/) (2019)
- [Nonparametric Density Estimation for High-Dimensional Data - Algorithms and Applications](https://mlsanity.com/paper/1904.00176/) (2019)
- [Learning Discrete Distributions by Dequantization](https://mlsanity.com/paper/2001.11235/) (2020)
- [Practical Evaluation of Out-of-Distribution Detection Methods for Image Classification](https://mlsanity.com/paper/2101.02447/) (2021)

## Built on by (most cited first, 15 of 15)

- [Unmasking the Chameleons: A Benchmark for Out-of-Distribution Detection in Medical Tabular Data](https://mlsanity.com/paper/2309.16220/) (2023)
- [Beyond AUROC & co. for evaluating out-of-distribution detection performance](https://mlsanity.com/paper/2306.14658/) (2023)
- [A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1](https://mlsanity.com/paper/2312.17506/) (2023)
- [Looking for Out-of-Distribution Environments in Multi-center Critical Care Data](https://mlsanity.com/paper/2205.13398/) (2022)
- [Model Monitoring and Robustness of In-Use Machine Learning Models: Quantifying Data Distribution Shifts Using Population Stability Index](https://mlsanity.com/paper/2302.00775/) (2023)
- [NCDD: Nearest Centroid Distance Deficit for Out-Of-Distribution Detection in Gastrointestinal Vision](https://mlsanity.com/paper/2412.01590/) (2024)
- [Towards Robust Deep Learning using Entropic Losses](https://mlsanity.com/paper/2208.03566/) (2022)
- [Revamping AI Models in Dermatology: Overcoming Critical Challenges for Enhanced Skin Lesion Diagnosis](https://mlsanity.com/paper/2311.01009/) (2023)
- [Reliably Detecting Model Failures in Deployment Without Labels](https://mlsanity.com/paper/2506.05047/) (2025)
- [DynaSubVAE: Adaptive Subgrouping for Scalable and Robust OOD Detection](https://mlsanity.com/paper/2506.10200/) (2025)
- [Influence of Classification Task and Distribution Shift Type on OOD Detection in Fetal Ultrasound](https://mlsanity.com/paper/2509.18326/) (2025)
- [General OOD Detection via Model-aware and Subspace-aware Variable Priority](https://mlsanity.com/paper/2512.13003/) (2025)
- [The Invisible Gorilla Effect in Out-of-distribution Detection](https://mlsanity.com/paper/2602.20068/) (2026)
- [Rethinking external validation for the target population: Capturing patient-level similarity with a generative model](https://mlsanity.com/paper/2605.11284/) (2026)
- [Training-Free Out-of-Distribution Detection for Pathology Whole-Slide Images](https://mlsanity.com/paper/2608.01407/) (2026)

## Abstract (arXiv)

Detection of Out-of-Distribution (OOD) samples in real time is a crucial safety check for deployment of machine learning models in the medical field. Despite a growing number of uncertainty quantification techniques, there is a lack of evaluation guidelines on how to select OOD detection methods in practice. This gap impedes implementation of OOD detection methods for real-world applications. Here, we propose a series of practical considerations and tests to choose the best OOD detector for a specific medical dataset. These guidelines are illustrated on a real-life use case of Electronic Health Records (EHR). Our results can serve as a guide for implementation of OOD detection methods in clinical practice, mitigating risks associated with the use of machine learning models in healthcare.