mlsanityData to 7 Oct 2026

Entropic Issues in Likelihood-Based OOD Detection

arXiv 2109.10794 ↗First posted 22 Sep 2021stat.ML cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Deep generative models trained by maximum likelihood remain very popular methods for reasoning about data probabilistically. However, it has been observed that they can assign higher likelihoods to out-of-distribution (OOD) data than in-distribution data, thus calling into question the meaning of these likelihood values. In this work we provide a novel perspective on this phenomenon, decomposing the average likelihood into a KL divergence term and an entropy term. We argue that the latter can explain the curious OOD behaviour mentioned above, suppressing likelihood values on datasets with higher entropy. Although our idea is simple, we have not seen it explored yet in the literature. This analysis provides further explanation for the success of OOD detection methods based on likelihood ratios, as the problematic entropy term cancels out in expectation. Finally, we discuss how this observation relates to recent success in OOD detection with manifold-supported models, for which the above decomposition does not hold directly.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

22 of 35 references.

References · 22

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  4. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  5. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  6. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  7. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  8. 2020Why Normalizing Flows Fail to Detect Out-of-Distribution Datacited by 220
  9. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  10. 2019Input complexity and out-of-distribution detection with likelihood-based generative modelscited by 186
  11. 2020Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encodercited by 145
  12. 2020Flows for simultaneous manifold learning and density estimationcited by 117
10 more
  1. 2021Understanding Failures in Out-of-Distribution Detection with Deep Generative Modelscited by 89
  2. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
  3. 2020Density of States Estimation for Out-of-Distribution Detectioncited by 74
  4. 2020Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Featurescited by 64
  5. 2021Rectangular Flows for Manifold Learningcited by 49
  6. 2020Perfect density models cannot guarantee anomaly detectioncited by 39
  7. 2016Linear dynamical neural population models through nonlinear embeddingscited by 36
  8. 2014Stochastic Backpropagation and Approximate Inference in Deep Generative Modelscited by 30
  9. 2019Implicit Generation and Generalization in Energy-Based Modelscited by 21
  10. 2019Deep Generative Models Strike Back! Improving Understanding and Evaluation in Light of Unmet Expectations for OoD Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 7; 2025: 2; 2026: 3.

16 papers.

Sort:

By area:

  1. 2022Diagnosing and Fixing Manifold Overfitting in Deep Generative Modelscited by 28
  2. 2022Augmenting Softmax Information for Selective Classification with Out-of-Distribution Datacited by 27
  3. 2024A Geometric Explanation of the Likelihood OOD Detection Paradoxcited by 22
  4. 2024Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detectorcited by 11
  5. 2024Forte : Finding Outliers with Representation Typicality Estimationcited by 9
  6. 2023Anomaly Detection with Variance Stabilized Density Estimationcited by 8
  7. 2022Neural Implicit Manifold Learning for Topology-Aware Density Estimationcited by 5
  8. 2024Can Your Generative Model Detect Out-of-Distribution Covariate Shift?cited by 4
  9. 2024Approximations to the Fisher Information Metric of Deep Generative Models for Out-Of-Distribution Detectioncited by 2
  10. 2024Differentiable Score-Based Likelihoods: Learning CT Motion Compensation From Clean Imagescited by 1
  11. 2024Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detectioncited by 0
  12. 2025From Lab to Factory: Pitfalls and Guidelines for Self-/Unsupervised Defect Detection on Low-Quality Industrial Imagescited by 0
4 more
  1. 2025Time-reversed Flow Matching with Worst Transport in High-dimensional Latent Space for Image Anomaly Detectioncited by 0
  2. 2026Mitigating the Likelihood Paradox in Flow-based OOD Detection via Entropy Manipulationcited by 0
  3. 2026Why the Counterintuitive Phenomenon of Likelihood Rarely Appears in Tabular Anomaly Detection with Deep Generative Models?cited by 0
  4. 2026The Invisible Gorilla Effect in Out-of-distribution Detectioncited by 0