# Understanding Failures in Out-of-Distribution Detection with Deep Generative Models

arXiv: https://arxiv.org/abs/2107.06908 · HTML: https://mlsanity.com/paper/2107.06908/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Why Normalizing Flows Fail to Detect Out-of-Distribution Data](https://mlsanity.com/paper/2006.08545/) (2020)
- [Input complexity and out-of-distribution detection with likelihood-based generative models](https://mlsanity.com/paper/1909.11480/) (2019)
- [PixelSNAIL: An Improved Autoregressive Generative Model](https://mlsanity.com/paper/1712.09763/) (2017)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)
- [Density of States Estimation for Out-of-Distribution Detection](https://mlsanity.com/paper/2006.09273/) (2020)
- [Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features](https://mlsanity.com/paper/2006.10848/) (2020)
- [Perfect density models cannot guarantee anomaly detection](https://mlsanity.com/paper/2012.03808/) (2020)
- [Further Analysis of Outlier Detection with Deep Generative Models](https://mlsanity.com/paper/2010.13064/) (2020)
- [Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence](https://mlsanity.com/paper/2106.15980/) (2021)

## Built on by (most cited first, 30 of 89)

- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Natural Synthetic Anomalies for Self-Supervised Anomaly Detection and Localization](https://mlsanity.com/paper/2109.15222/) (2021)
- [Is Out-of-Distribution Detection Learnable?](https://mlsanity.com/paper/2210.14707/) (2022)
- [Denoising diffusion models for out-of-distribution detection](https://mlsanity.com/paper/2211.07740/) (2022)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [Boosting Out-of-distribution Detection with Typical Features](https://mlsanity.com/paper/2210.04200/) (2022)
- [Raising the Bar in Graph-level Anomaly Detection](https://mlsanity.com/paper/2205.13845/) (2022)
- [DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition](https://mlsanity.com/paper/2207.02606/) (2022)
- [Learning to Augment Distributions for Out-of-Distribution Detection](https://mlsanity.com/paper/2311.01796/) (2023)
- [Maskomaly:Zero-Shot Mask Anomaly Segmentation](https://mlsanity.com/paper/2305.16972/) (2023)
- [Watermarking for Out-of-distribution Detection](https://mlsanity.com/paper/2210.15198/) (2022)
- [Out-of-distribution Detection via Frequency-regularized Generative Models](https://mlsanity.com/paper/2208.09083/) (2022)
- [ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection](https://mlsanity.com/paper/2402.17888/) (2024)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Improving the Intra-class Long-tail in 3D Detection via Rare Example Mining](https://mlsanity.com/paper/2210.08375/) (2022)
- [A Learning Based Hypothesis Test for Harmful Covariate Shift](https://mlsanity.com/paper/2212.02742/) (2022)
- [Entropic Issues in Likelihood-Based OOD Detection](https://mlsanity.com/paper/2109.10794/) (2021)
- [Back to the Basics: Revisiting Out-of-Distribution Detection Baselines](https://mlsanity.com/paper/2207.03061/) (2022)
- [Out-of-Distribution Detection for Medical Applications: Guidelines for Practical Evaluation](https://mlsanity.com/paper/2109.14885/) (2021)
- [Model-agnostic out-of-distribution detection using combined statistical tests](https://mlsanity.com/paper/2203.01097/) (2022)
- [Detecting Out-Of-Distribution Earth Observation Images with Diffusion Models](https://mlsanity.com/paper/2404.12667/) (2024)
- [Out-of-Distribution Detection Methods Answer the Wrong Questions](https://mlsanity.com/paper/2507.01831/) (2025)
- [Star-Shaped Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2302.05259/) (2023)
- [Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models](https://mlsanity.com/paper/2410.08611/) (2024)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [Data-SUITE: Data-centric identification of in-distribution incongruous examples](https://mlsanity.com/paper/2202.08836/) (2022)
- [Out-of-Distribution Detection with Reconstruction Error and Typicality-based Penalty](https://mlsanity.com/paper/2212.12641/) (2022)
- [Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data](https://mlsanity.com/paper/2112.12833/) (2021)
- [Normalizing Flow Ensembles for Rich Aleatoric and Epistemic Uncertainty Modeling](https://mlsanity.com/paper/2302.01312/) (2023)
- [OOD-Speech: A Large Bengali Speech Recognition Dataset for Out-of-Distribution Benchmarking](https://mlsanity.com/paper/2305.09688/) (2023)

## Abstract (arXiv)

Deep generative models (DGMs) seem a natural fit for detecting out-of-distribution (OOD) inputs, but such models have been shown to assign higher probabilities or densities to OOD images than images from the training distribution. In this work, we explain why this behavior should be attributed to model misestimation. We first prove that no method can guarantee performance beyond random chance without assumptions on which out-distributions are relevant. We then interrogate the typical set hypothesis, the claim that relevant out-distributions can lie in high likelihood regions of the data distribution, and that OOD detection should be defined based on the data distribution's typical set. We highlight the consequences implied by assuming support overlap between in- and out-distributions, as well as the arbitrariness of the typical set for OOD detection. Our results suggest that estimation error is a more plausible explanation than the misalignment between likelihood-based OOD detection and out-distributions of interest, and we illustrate how even minimal estimation error can lead to OOD detection failures, yielding implications for future work in deep generative modeling and OOD detection.