# Learning Probabilistic Models from Generator Latent Spaces with Hat EBM

arXiv: https://arxiv.org/abs/2210.16486 · HTML: https://mlsanity.com/paper/2210.16486/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 29 Oct 2022.

## Builds on (most cited first, 30 of 34)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [Improved Contrastive Divergence Training of Energy Based Models](https://mlsanity.com/paper/2012.01316/) (2020)
- [On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models](https://mlsanity.com/paper/1903.12370/) (2019)
- [Learning Latent Space Energy-Based Prior Model](https://mlsanity.com/paper/2006.08205/) (2020)
- [VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models](https://mlsanity.com/paper/2010.00654/) (2020)
- [Self-Supervised GANs via Auxiliary Rotation Loss](https://mlsanity.com/paper/1811.11212/) (2018)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [Generalized Energy Based Models](https://mlsanity.com/paper/2003.05033/) (2020)
- [Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling](https://mlsanity.com/paper/2003.06060/) (2020)
- [No MCMC for me: Amortized sampling for fast and stable training of energy-based models](https://mlsanity.com/paper/2010.04230/) (2020)
- [Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models](https://mlsanity.com/paper/2005.13525/) (2020)
- [Learning Descriptor Networks for 3D Shape Synthesis and Analysis](https://mlsanity.com/paper/1804.00586/) (2018)
- [Maximum Entropy Generators for Energy-Based Models](https://mlsanity.com/paper/1901.08508/) (2019)
- [Refining Deep Generative Models via Discriminator Gradient Flow](https://mlsanity.com/paper/2012.00780/) (2020)
- [InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning](https://mlsanity.com/paper/2007.04589/) (2020)
- [Deep Directed Generative Models with Energy-Based Probability Estimation](https://mlsanity.com/paper/1606.03439/) (2016)
- [Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model](https://mlsanity.com/paper/1812.10907/) (2018)
- [Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Sampling](https://mlsanity.com/paper/1709.08868/) (2017)
- [Discriminator optimal transport](https://mlsanity.com/paper/1910.06832/) (2019)

## Built on by (most cited first, 11 of 11)

- [You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs](https://mlsanity.com/paper/2403.12931/) (2024)
- [Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood](https://mlsanity.com/paper/2309.05153/) (2023)
- [EGC: Image Generation and Classification via a Diffusion Energy-Based Model](https://mlsanity.com/paper/2304.02012/) (2023)
- [Learning Energy-based Model via Dual-MCMC Teaching](https://mlsanity.com/paper/2312.02469/) (2023)
- [Improving Adversarial Energy-Based Model via Diffusion Process](https://mlsanity.com/paper/2403.01666/) (2024)
- [Learning Latent Space Hierarchical EBM Diffusion Models](https://mlsanity.com/paper/2405.13910/) (2024)
- [Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detection](https://mlsanity.com/paper/2501.16718/) (2025)
- [Energy-Calibrated VAE with Test Time Free Lunch](https://mlsanity.com/paper/2311.04071/) (2023)
- [Learning Energy-Based Generative Models via Potential Flow: A Variational Principle Approach to Probability Density Homotopy Matching](https://mlsanity.com/paper/2504.16262/) (2025)
- [Variational Potential Flow: A Novel Probabilistic Framework for Energy-Based Generative Modelling](https://mlsanity.com/paper/2407.15238/) (2024)
- [Annealed Langevin Posterior Sampling (ALPS): A Rapid Algorithm for Image Restoration with Multiscale Energy Models](https://mlsanity.com/paper/2601.02594/) (2026)

## Abstract (arXiv)

This work proposes a method for using any generator network as the foundation of an Energy-Based Model (EBM). Our formulation posits that observed images are the sum of unobserved latent variables passed through the generator network and a residual random variable that spans the gap between the generator output and the image manifold. One can then define an EBM that includes the generator as part of its forward pass, which we call the Hat EBM. The model can be trained without inferring the latent variables of the observed data or calculating the generator Jacobian determinant. This enables explicit probabilistic modeling of the output distribution of any type of generator network. Experiments show strong performance of the proposed method on (1) unconditional ImageNet synthesis at 128x128 resolution, (2) refining the output of existing generators, and (3) learning EBMs that incorporate non-probabilistic generators. Code and pretrained models to reproduce our results are available at https://github.com/point0bar1/hat-ebm.