# Entropic Issues in Likelihood-Based OOD Detection

arXiv: https://arxiv.org/abs/2109.10794 · HTML: https://mlsanity.com/paper/2109.10794/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Why Normalizing Flows Fail to Detect Out-of-Distribution Data](https://mlsanity.com/paper/2006.08545/) (2020)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Input complexity and out-of-distribution detection with likelihood-based generative models](https://mlsanity.com/paper/1909.11480/) (2019)
- [Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder](https://mlsanity.com/paper/2003.02977/) (2020)
- [Flows for simultaneous manifold learning and density estimation](https://mlsanity.com/paper/2003.13913/) (2020)
- [Understanding Failures in Out-of-Distribution Detection with Deep Generative Models](https://mlsanity.com/paper/2107.06908/) (2021)
- [Practical Lossless Compression with Latent Variables using Bits Back Coding](https://mlsanity.com/paper/1901.04866/) (2019)
- [Density of States Estimation for Out-of-Distribution Detection](https://mlsanity.com/paper/2006.09273/) (2020)
- [Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features](https://mlsanity.com/paper/2006.10848/) (2020)
- [Rectangular Flows for Manifold Learning](https://mlsanity.com/paper/2106.01413/) (2021)
- [Perfect density models cannot guarantee anomaly detection](https://mlsanity.com/paper/2012.03808/) (2020)
- [Linear dynamical neural population models through nonlinear embeddings](https://mlsanity.com/paper/1605.08454/) (2016)
- [Stochastic Backpropagation and Approximate Inference in Deep Generative Models](https://mlsanity.com/paper/1401.4082/) (2014)
- [Implicit Generation and Generalization in Energy-Based Models](https://mlsanity.com/paper/1903.08689/) (2019)
- [Deep Generative Models Strike Back! Improving Understanding and Evaluation in Light of Unmet Expectations for OoD Data](https://mlsanity.com/paper/1911.04699/) (2019)

## Built on by (most cited first, 16 of 16)

- [Diagnosing and Fixing Manifold Overfitting in Deep Generative Models](https://mlsanity.com/paper/2204.07172/) (2022)
- [Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data](https://mlsanity.com/paper/2207.07506/) (2022)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector](https://mlsanity.com/paper/2404.08679/) (2024)
- [Forte : Finding Outliers with Representation Typicality Estimation](https://mlsanity.com/paper/2410.01322/) (2024)
- [Anomaly Detection with Variance Stabilized Density Estimation](https://mlsanity.com/paper/2306.00582/) (2023)
- [Neural Implicit Manifold Learning for Topology-Aware Density Estimation](https://mlsanity.com/paper/2206.11267/) (2022)
- [Can Your Generative Model Detect Out-of-Distribution Covariate Shift?](https://mlsanity.com/paper/2409.03043/) (2024)
- [Approximations to the Fisher Information Metric of Deep Generative Models for Out-Of-Distribution Detection](https://mlsanity.com/paper/2403.01485/) (2024)
- [Differentiable Score-Based Likelihoods: Learning CT Motion Compensation From Clean Images](https://mlsanity.com/paper/2404.14747/) (2024)
- [Time-reversed Flow Matching with Worst Transport in High-dimensional Latent Space for Image Anomaly Detection](https://mlsanity.com/paper/2508.05461/) (2025)
- [Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2409.03801/) (2024)
- [From Lab to Factory: Pitfalls and Guidelines for Self-/Unsupervised Defect Detection on Low-Quality Industrial Images](https://mlsanity.com/paper/2506.16890/) (2025)
- [Mitigating the Likelihood Paradox in Flow-based OOD Detection via Entropy Manipulation](https://mlsanity.com/paper/2602.09581/) (2026)
- [Why the Counterintuitive Phenomenon of Likelihood Rarely Appears in Tabular Anomaly Detection with Deep Generative Models?](https://mlsanity.com/paper/2602.09593/) (2026)
- [The Invisible Gorilla Effect in Out-of-distribution Detection](https://mlsanity.com/paper/2602.20068/) (2026)

## Abstract (arXiv)

Deep generative models trained by maximum likelihood remain very popular methods for reasoning about data probabilistically. However, it has been observed that they can assign higher likelihoods to out-of-distribution (OOD) data than in-distribution data, thus calling into question the meaning of these likelihood values. In this work we provide a novel perspective on this phenomenon, decomposing the average likelihood into a KL divergence term and an entropy term. We argue that the latter can explain the curious OOD behaviour mentioned above, suppressing likelihood values on datasets with higher entropy. Although our idea is simple, we have not seen it explored yet in the literature. This analysis provides further explanation for the success of OOD detection methods based on likelihood ratios, as the problematic entropy term cancels out in expectation. Finally, we discuss how this observation relates to recent success in OOD detection with manifold-supported models, for which the above decomposition does not hold directly.