# Autoencoding Under Normalization Constraints

arXiv: https://arxiv.org/abs/2105.05735 · HTML: https://mlsanity.com/paper/2105.05735/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 68 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training](https://mlsanity.com/paper/1805.06725/) (2018)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Improving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders](https://mlsanity.com/paper/1807.02011/) (2018)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Input complexity and out-of-distribution detection with likelihood-based generative models](https://mlsanity.com/paper/1909.11480/) (2019)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder](https://mlsanity.com/paper/2003.02977/) (2020)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [Energy-based Generative Adversarial Network](https://mlsanity.com/paper/1609.03126/) (2016)
- [Flows for simultaneous manifold learning and density estimation](https://mlsanity.com/paper/2003.13913/) (2020)
- [VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models](https://mlsanity.com/paper/2010.00654/) (2020)
- [Generative Probabilistic Novelty Detection with Adversarial Autoencoders](https://mlsanity.com/paper/1807.02588/) (2018)

## Built on by (most cited first, 26 of 26)

- [Mask, Stitch, and Re-Sample: Enhancing Robustness and Generalizability in Anomaly Detection through Automatic Diffusion Models](https://mlsanity.com/paper/2305.19643/) (2023)
- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [Machine Learning for Anomaly Detection in Particle Physics](https://mlsanity.com/paper/2312.14190/) (2023)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach](https://mlsanity.com/paper/2310.18677/) (2023)
- [A Review of Change of Variable Formulas for Generative Modeling](https://mlsanity.com/paper/2308.02652/) (2023)
- [Autoencoders for Anomaly Detection are Unreliable](https://mlsanity.com/paper/2501.13864/) (2025)
- ["Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood](https://mlsanity.com/paper/2405.16730/) (2024)
- [Normality-Calibrated Autoencoder for Unsupervised Anomaly Detection on Data Contamination](https://mlsanity.com/paper/2110.14825/) (2021)
- [Data Invariants to Understand Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2111.13362/) (2021)
- [Neural Implicit Manifold Learning for Topology-Aware Density Estimation](https://mlsanity.com/paper/2206.11267/) (2022)
- [MMP++: Motion Manifold Primitives with Parametric Curve Models](https://mlsanity.com/paper/2310.17072/) (2023)
- [An Iterative Method for Unsupervised Robust Anomaly Detection Under Data Contamination](https://mlsanity.com/paper/2309.09436/) (2023)
- [Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models](https://mlsanity.com/paper/2407.00626/) (2024)
- [Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Data](https://mlsanity.com/paper/2208.14024/) (2022)
- [Versatile Energy-Based Probabilistic Models for High Energy Physics](https://mlsanity.com/paper/2302.00695/) (2023)
- [TR0N: Translator Networks for 0-Shot Plug-and-Play Conditional Generation](https://mlsanity.com/paper/2304.13742/) (2023)
- [Restricted Generative Projection for One-Class Classification and Anomaly Detection](https://mlsanity.com/paper/2307.04097/) (2023)
- [Deep Positive-Unlabeled Anomaly Detection for Contaminated Unlabeled Data](https://mlsanity.com/paper/2405.18929/) (2024)
- [Geometric instability of out of distribution data across autoencoder architecture](https://mlsanity.com/paper/2201.11902/) (2022)
- [Evaluating Out-of-Distribution Detectors Through Adversarial Generation of Outliers](https://mlsanity.com/paper/2208.10940/) (2022)
- [How to Construct Energy for Images? Denoising Autoencoder Can Be Energy Based Model](https://mlsanity.com/paper/2303.03887/) (2023)
- [Triggering Dark Showers with Conditional Dual Auto-Encoders](https://mlsanity.com/paper/2306.12955/) (2023)
- [Learning Nonparametric High-Dimensional Generative Models: The Empirical-Beta-Copula Autoencoder](https://mlsanity.com/paper/2309.09916/) (2023)
- [Adversarial Anomaly Detection using Gaussian Priors and Nonlinear Anomaly Scores](https://mlsanity.com/paper/2310.18091/) (2023)
- [Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning](https://mlsanity.com/paper/2312.03397/) (2023)

## Abstract (arXiv)

Likelihood is a standard estimate for outlier detection. The specific role of the normalization constraint is to ensure that the out-of-distribution (OOD) regime has a small likelihood when samples are learned using maximum likelihood. Because autoencoders do not possess such a process of normalization, they often fail to recognize outliers even when they are obviously OOD. We propose the Normalized Autoencoder (NAE), a normalized probabilistic model constructed from an autoencoder. The probability density of NAE is defined using the reconstruction error of an autoencoder, which is differently defined in the conventional energy-based model. In our model, normalization is enforced by suppressing the reconstruction of negative samples, significantly improving the outlier detection performance. Our experimental results confirm the efficacy of NAE, both in detecting outliers and in generating in-distribution samples.