# Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features

arXiv: https://arxiv.org/abs/2006.10848 · HTML: https://mlsanity.com/paper/2006.10848/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 30 of 64)

- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows](https://mlsanity.com/paper/2107.12571/) (2021)
- [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022)
- [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021)
- [Understanding Failures in Out-of-Distribution Detection with Deep Generative Models](https://mlsanity.com/paper/2107.06908/) (2021)
- [Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild](https://mlsanity.com/paper/2203.03800/) (2022)
- [Hierarchical VAEs Know What They Don't Know](https://mlsanity.com/paper/2102.08248/) (2021)
- [RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection](https://mlsanity.com/paper/2209.08590/) (2022)
- [On the Out-of-distribution Generalization of Probabilistic Image Modelling](https://mlsanity.com/paper/2109.02639/) (2021)
- [Autoencoders for unsupervised anomaly detection in high energy physics](https://mlsanity.com/paper/2104.09051/) (2021)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Reversing the Abnormal: Pseudo-Healthy Generative Networks for Anomaly Detection](https://mlsanity.com/paper/2303.08452/) (2023)
- [On Out-of-distribution Detection with Energy-based Models](https://mlsanity.com/paper/2107.08785/) (2021)
- [Entropic Issues in Likelihood-Based OOD Detection](https://mlsanity.com/paper/2109.10794/) (2021)
- [Learning Sample Difficulty from Pre-trained Models for Reliable Prediction](https://mlsanity.com/paper/2304.10127/) (2023)
- [GOLD: Graph Out-of-Distribution Detection via Implicit Adversarial Latent Generation](https://mlsanity.com/paper/2502.05780/) (2025)
- [Model-agnostic out-of-distribution detection using combined statistical tests](https://mlsanity.com/paper/2203.01097/) (2022)
- [Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector](https://mlsanity.com/paper/2404.08679/) (2024)
- [Falsehoods that ML researchers believe about OOD detection](https://mlsanity.com/paper/2210.12767/) (2022)
- [Taming False Positives in Out-of-Distribution Detection with Human Feedback](https://mlsanity.com/paper/2404.16954/) (2024)
- [New Methods and Datasets for Group Anomaly Detection From Fundamental Physics](https://mlsanity.com/paper/2107.02821/) (2021)
- [Improving VAE-based Representation Learning](https://mlsanity.com/paper/2205.14539/) (2022)
- [What do we learn? Debunking the Myth of Unsupervised Outlier Detection](https://mlsanity.com/paper/2206.03698/) (2022)
- [Joint Lossless Compression and Steganography for Medical Images via Large Language Models](https://mlsanity.com/paper/2508.01782/) (2025)
- [Data Invariants to Understand Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2111.13362/) (2021)
- [Out of Distribution Detection on ImageNet-O](https://mlsanity.com/paper/2201.09352/) (2022)
- [PILC: Practical Image Lossless Compression with an End-to-end GPU Oriented Neural Framework](https://mlsanity.com/paper/2206.05279/) (2022)
- [Understanding Likelihood of Normalizing Flow and Image Complexity through the Lens of Out-of-Distribution Detection](https://mlsanity.com/paper/2402.10477/) (2024)
- [MultiMax: Sparse and Multi-Modal Attention Learning](https://mlsanity.com/paper/2406.01189/) (2024)

## Abstract (arXiv)

Deep generative networks trained via maximum likelihood on a natural image dataset like CIFAR10 often assign high likelihoods to images from datasets with different objects (e.g., SVHN). We refine previous investigations of this failure at anomaly detection for invertible generative networks and provide a clear explanation of it as a combination of model bias and domain prior: Convolutional networks learn similar low-level feature distributions when trained on any natural image dataset and these low-level features dominate the likelihood. Hence, when the discriminative features between inliers and outliers are on a high-level, e.g., object shapes, anomaly detection becomes particularly challenging. To remove the negative impact of model bias and domain prior on detecting high-level differences, we propose two methods, first, using the log likelihood ratios of two identical models, one trained on the in-distribution data (e.g., CIFAR10) and the other one on a more general distribution of images (e.g., 80 Million Tiny Images). We also derive a novel outlier loss for the in-distribution network on samples from the more general distribution to further improve the performance. Secondly, using a multi-scale model like Glow, we show that low-level features are mainly captured at early scales. Therefore, using only the likelihood contribution of the final scale performs remarkably well for detecting high-level feature differences of the out-of-distribution and the in-distribution. This method is especially useful if one does not have access to a suitable general distribution. Overall, our methods achieve strong anomaly detection performance in the unsupervised setting, and only slightly underperform state-of-the-art classifier-based methods in the supervised setting. Code can be found at https://github.com/boschresearch/hierarchical_anomaly_detection.