# Maximum Entropy Generators for Energy-Based Models

arXiv: https://arxiv.org/abs/1901.08508 · HTML: https://mlsanity.com/paper/1901.08508/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2019.

## Built on by (most cited first, 30 of 56)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [GFlowNet Foundations](https://mlsanity.com/paper/2111.09266/) (2021)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds](https://mlsanity.com/paper/2109.00179/) (2021)
- [Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement](https://mlsanity.com/paper/2301.03028/) (2023)
- [Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement](https://mlsanity.com/paper/1910.04417/) (2019)
- [Towards Understanding the Generative Capability of Adversarially Robust Classifiers](https://mlsanity.com/paper/2108.09093/) (2021)
- [Energy-guided Entropic Neural Optimal Transport](https://mlsanity.com/paper/2304.06094/) (2023)
- [Learning Implicit Priors for Motion Optimization](https://mlsanity.com/paper/2204.05369/) (2022)
- [Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood](https://mlsanity.com/paper/2309.05153/) (2023)
- [Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model](https://mlsanity.com/paper/2209.08739/) (2022)
- [Guiding Energy-based Models via Contrastive Latent Variables](https://mlsanity.com/paper/2303.03023/) (2023)
- [Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach](https://mlsanity.com/paper/2310.18677/) (2023)
- [EGC: Image Generation and Classification via a Diffusion Energy-Based Model](https://mlsanity.com/paper/2304.02012/) (2023)
- [A survey and taxonomy of loss functions in machine learning](https://mlsanity.com/paper/2301.05579/) (2023)
- [Learning Energy-Based Prior Model with Diffusion-Amortized MCMC](https://mlsanity.com/paper/2310.03218/) (2023)
- [MemStream: Memory-Based Streaming Anomaly Detection](https://mlsanity.com/paper/2106.03837/) (2021)
- [Explaining the effects of non-convergent sampling in the training of Energy-Based Models](https://mlsanity.com/paper/2301.09428/) (2023)
- [Learning Probabilistic Models from Generator Latent Spaces with Hat EBM](https://mlsanity.com/paper/2210.16486/) (2022)
- [Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent](https://mlsanity.com/paper/2106.07832/) (2021)
- [Bounds all around: training energy-based models with bidirectional bounds](https://mlsanity.com/paper/2111.00929/) (2021)
- [Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models](https://mlsanity.com/paper/2309.05803/) (2023)
- [Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling](https://mlsanity.com/paper/2111.02434/) (2021)
- [Non-Imaging Medical Data Synthesis for Trustworthy AI: A Comprehensive Survey](https://mlsanity.com/paper/2209.09239/) (2022)
- [Energy Discrepancies: A Score-Independent Loss for Energy-Based Models](https://mlsanity.com/paper/2307.06431/) (2023)
- [Time-series Generation by Contrastive Imitation](https://mlsanity.com/paper/2311.01388/) (2023)
- [MRI Reconstruction Using Deep Energy-Based Model](https://mlsanity.com/paper/2109.03237/) (2021)
- [Learning Joint Latent Space EBM Prior Model for Multi-layer Generator](https://mlsanity.com/paper/2306.06323/) (2023)
- [Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting](https://mlsanity.com/paper/2306.14902/) (2023)

## Abstract (arXiv)

Maximum likelihood estimation of energy-based models is a challenging problem due to the intractability of the log-likelihood gradient. In this work, we propose learning both the energy function and an amortized approximate sampling mechanism using a neural generator network, which provides an efficient approximation of the log-likelihood gradient. The resulting objective requires maximizing entropy of the generated samples, which we perform using recently proposed nonparametric mutual information estimators. Finally, to stabilize the resulting adversarial game, we use a zero-centered gradient penalty derived as a necessary condition from the score matching literature. The proposed technique can generate sharp images with Inception and FID scores competitive with recent GAN techniques, does not suffer from mode collapse, and is competitive with state-of-the-art anomaly detection techniques.