# Hierarchical VAEs Know What They Don't Know

arXiv: https://arxiv.org/abs/2102.08248 · HTML: https://mlsanity.com/paper/2102.08248/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 64 references that are in the map.
- First posted on arXiv 16 Feb 2021.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)

## Built on by (most cited first, 30 of 58)

- [Denoising diffusion models for out-of-distribution detection](https://mlsanity.com/paper/2211.07740/) (2022)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Conditional Image Generation by Conditioning Variational Auto-Encoders](https://mlsanity.com/paper/2102.12037/) (2021)
- [Unsupervised 3D out-of-distribution detection with latent diffusion models](https://mlsanity.com/paper/2307.03777/) (2023)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model](https://mlsanity.com/paper/2209.08739/) (2022)
- [t3-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence](https://mlsanity.com/paper/2312.01133/) (2023)
- [Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?](https://mlsanity.com/paper/2105.09270/) (2021)
- [Model-agnostic out-of-distribution detection using combined statistical tests](https://mlsanity.com/paper/2203.01097/) (2022)
- [NPCL: Neural Processes for Uncertainty-Aware Continual Learning](https://mlsanity.com/paper/2310.19272/) (2023)
- [Bounds all around: training energy-based models with bidirectional bounds](https://mlsanity.com/paper/2111.00929/) (2021)
- [Likelihood-based Out-of-Distribution Detection with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2310.17432/) (2023)
- [Autoencoders for Anomaly Detection are Unreliable](https://mlsanity.com/paper/2501.13864/) (2025)
- [Learning Joint Latent Space EBM Prior Model for Multi-layer Generator](https://mlsanity.com/paper/2306.06323/) (2023)
- [How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agents](https://mlsanity.com/paper/2110.03608/) (2021)
- [eVAE: Evolutionary Variational Autoencoder](https://mlsanity.com/paper/2301.00011/) (2023)
- [Inverse problem regularization with hierarchical variational autoencoders](https://mlsanity.com/paper/2303.11217/) (2023)
- [Learning Unnormalized Statistical Models via Compositional Optimization](https://mlsanity.com/paper/2306.07485/) (2023)
- [Learning Hierarchical Features with Joint Latent Space Energy-Based Prior](https://mlsanity.com/paper/2310.09604/) (2023)
- [Learning Latent Space Hierarchical EBM Diffusion Models](https://mlsanity.com/paper/2405.13910/) (2024)
- [Data Invariants to Understand Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2111.13362/) (2021)
- [Parallel Neural Local Lossless Compression](https://mlsanity.com/paper/2201.05213/) (2022)
- [When less is more: Simplifying inputs aids neural network understanding](https://mlsanity.com/paper/2201.05610/) (2022)
- [Out-of-Distribution Detection with Class Ratio Estimation](https://mlsanity.com/paper/2206.03955/) (2022)
- [Feature Density Estimation for Out-of-Distribution Detection via Normalizing Flows](https://mlsanity.com/paper/2402.06537/) (2024)
- [A Non-negative VAE:the Generalized Gamma Belief Network](https://mlsanity.com/paper/2408.03388/) (2024)
- [Robust uncertainty estimates with out-of-distribution pseudo-inputs training](https://mlsanity.com/paper/2201.05890/) (2022)
- [How to train your VAE](https://mlsanity.com/paper/2309.13160/) (2023)
- [Robust Out-of-Distribution Detection on Deep Probabilistic Generative Models](https://mlsanity.com/paper/2106.07903/) (2021)
- [Top-down inference in an early visual cortex inspired hierarchical Variational Autoencoder](https://mlsanity.com/paper/2206.00436/) (2022)

## Abstract (arXiv)

Deep generative models have been demonstrated as state-of-the-art density estimators. Yet, recent work has found that they often assign a higher likelihood to data from outside the training distribution. This seemingly paradoxical behavior has caused concerns over the quality of the attained density estimates. In the context of hierarchical variational autoencoders, we provide evidence to explain this behavior by out-of-distribution data having in-distribution low-level features. We argue that this is both expected and desirable behavior. With this insight in hand, we develop a fast, scalable and fully unsupervised likelihood-ratio score for OOD detection that requires data to be in-distribution across all feature-levels. We benchmark the method on a vast set of data and model combinations and achieve state-of-the-art results on out-of-distribution detection.