# Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler

arXiv: https://arxiv.org/abs/2012.14936 · HTML: https://mlsanity.com/paper/2012.14936/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 30 of 43)

- [Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC](https://mlsanity.com/paper/2302.11552/) (2023)
- [Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction](https://mlsanity.com/paper/2112.13528/) (2021)
- [Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning](https://mlsanity.com/paper/2206.08321/) (2022)
- [Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation](https://mlsanity.com/paper/2207.00787/) (2022)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)
- [You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs](https://mlsanity.com/paper/2403.12931/) (2024)
- [Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood](https://mlsanity.com/paper/2309.05153/) (2023)
- [S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic](https://mlsanity.com/paper/2405.00987/) (2024)
- [EGC: Image Generation and Classification via a Diffusion Energy-Based Model](https://mlsanity.com/paper/2304.02012/) (2023)
- [Efficient Training of Energy-Based Models Using Jarzynski Equality](https://mlsanity.com/paper/2305.19414/) (2023)
- [Learning Energy-Based Prior Model with Diffusion-Amortized MCMC](https://mlsanity.com/paper/2310.03218/) (2023)
- [Bounds all around: training energy-based models with bidirectional bounds](https://mlsanity.com/paper/2111.00929/) (2021)
- [Hybrid Energy Based Model in the Feature Space for Out-of-Distribution Detection](https://mlsanity.com/paper/2305.16966/) (2023)
- [Energy-based Potential Games for Joint Motion Forecasting and Control](https://mlsanity.com/paper/2312.01811/) (2023)
- [Decoupled Graph Energy-based Model for Node Out-of-Distribution Detection on Heterophilic Graphs](https://mlsanity.com/paper/2502.17912/) (2025)
- [Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation](https://mlsanity.com/paper/2103.04285/) (2021)
- [Energy-Based Generative Cooperative Saliency Prediction](https://mlsanity.com/paper/2106.13389/) (2021)
- [Multimeasurement Generative Models](https://mlsanity.com/paper/2112.09822/) (2021)
- [Learning Energy-based Model via Dual-MCMC Teaching](https://mlsanity.com/paper/2312.02469/) (2023)
- [Improving Adversarial Energy-Based Model via Diffusion Process](https://mlsanity.com/paper/2403.01666/) (2024)
- [Bi-level Doubly Variational Learning for Energy-based Latent Variable Models](https://mlsanity.com/paper/2203.14702/) (2022)
- [Learning Unnormalized Statistical Models via Compositional Optimization](https://mlsanity.com/paper/2306.07485/) (2023)
- [Learning Latent Space Hierarchical EBM Diffusion Models](https://mlsanity.com/paper/2405.13910/) (2024)
- [Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamics](https://mlsanity.com/paper/2510.12311/) (2025)
- [Balanced Training of Energy-Based Models with Adaptive Flow Sampling](https://mlsanity.com/paper/2306.00684/) (2023)
- [On a Connection between Differential Games, Optimal Control, and Energy-based Models for Multi-Agent Interactions](https://mlsanity.com/paper/2308.16539/) (2023)
- [Learning Proposals for Practical Energy-Based Regression](https://mlsanity.com/paper/2110.11948/) (2021)
- [An Energy-Based Prior for Generative Saliency](https://mlsanity.com/paper/2204.08803/) (2022)
- [Energy-Calibrated VAE with Test Time Free Lunch](https://mlsanity.com/paper/2311.04071/) (2023)
- [EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss](https://mlsanity.com/paper/2102.11757/) (2021)

## Abstract (arXiv)

Due to the intractable partition function, training energy-based models (EBMs) by maximum likelihood requires Markov chain Monte Carlo (MCMC) sampling to approximate the gradient of the Kullback-Leibler divergence between data and model distributions. However, it is non-trivial to sample from an EBM because of the difficulty of mixing between modes. In this paper, we propose to learn a variational auto-encoder (VAE) to initialize the finite-step MCMC, such as Langevin dynamics that is derived from the energy function, for efficient amortized sampling of the EBM. With these amortized MCMC samples, the EBM can be trained by maximum likelihood, which follows an "analysis by synthesis" scheme; while the VAE learns from these MCMC samples via variational Bayes. We call this joint training algorithm the variational MCMC teaching, in which the VAE chases the EBM toward data distribution. We interpret the learning algorithm as a dynamic alternating projection in the context of information geometry. Our proposed models can generate samples comparable to GANs and EBMs. Additionally, we demonstrate that our model can learn effective probabilistic distribution toward supervised conditional learning tasks.