# Joint Training of Variational Auto-Encoder and Latent Energy-Based Model

arXiv: https://arxiv.org/abs/2006.06059 · HTML: https://mlsanity.com/paper/2006.06059/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 30 of 30)

- [Latent Diffusion Energy-Based Model for Interpretable Text Modeling](https://mlsanity.com/paper/2206.05895/) (2022)
- [Trajectory Prediction with Latent Belief Energy-Based Model](https://mlsanity.com/paper/2104.03086/) (2021)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)
- [JEM++: Improved Techniques for Training JEM](https://mlsanity.com/paper/2109.09032/) (2021)
- [Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood](https://mlsanity.com/paper/2309.05153/) (2023)
- [Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach](https://mlsanity.com/paper/2310.18677/) (2023)
- [Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classification](https://mlsanity.com/paper/2108.11556/) (2021)
- [Learning Joint Latent Space EBM Prior Model for Multi-layer Generator](https://mlsanity.com/paper/2306.06323/) (2023)
- ["Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood](https://mlsanity.com/paper/2405.16730/) (2024)
- [A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference](https://mlsanity.com/paper/2301.09300/) (2023)
- [Learning Energy-based Model via Dual-MCMC Teaching](https://mlsanity.com/paper/2312.02469/) (2023)
- [Improving Adversarial Energy-Based Model via Diffusion Process](https://mlsanity.com/paper/2403.01666/) (2024)
- [Learning Multimodal Latent Generative Models with Energy-Based Prior](https://mlsanity.com/paper/2409.19862/) (2024)
- [Elsa: Energy-based learning for semi-supervised anomaly detection](https://mlsanity.com/paper/2103.15296/) (2021)
- [Bi-level Doubly Variational Learning for Energy-based Latent Variable Models](https://mlsanity.com/paper/2203.14702/) (2022)
- [Learning Latent Space Hierarchical EBM Diffusion Models](https://mlsanity.com/paper/2405.13910/) (2024)
- [EQ-CBM: A Probabilistic Concept Bottleneck with Energy-based Models and Quantized Vectors](https://mlsanity.com/paper/2409.14630/) (2024)
- [Energy-Calibrated VAE with Test Time Free Lunch](https://mlsanity.com/paper/2311.04071/) (2023)
- [Conjugate Energy-Based Models](https://mlsanity.com/paper/2106.13798/) (2021)
- [Learning Energy-based Variational Latent Prior for VAEs](https://mlsanity.com/paper/2510.00260/) (2025)
- [How to Construct Energy for Images? Denoising Autoencoder Can Be Energy Based Model](https://mlsanity.com/paper/2303.03887/) (2023)
- [Learning Multimodal Latent Space with EBM Prior and MCMC Inference](https://mlsanity.com/paper/2408.10467/) (2024)
- [Synergizing Transport-Based Generative Models and Latent Geometry for Stochastic Closure Modeling](https://mlsanity.com/paper/2602.17089/) (2026)
- [Deep Consensus Learning](https://mlsanity.com/paper/2103.08475/) (2021)
- [STANLEY: Stochastic Gradient Anisotropic Langevin Dynamics for Learning Energy-Based Models](https://mlsanity.com/paper/2310.12667/) (2023)
- [Variational Potential Flow: A Novel Probabilistic Framework for Energy-Based Generative Modelling](https://mlsanity.com/paper/2407.15238/) (2024)
- [A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks](https://mlsanity.com/paper/2409.01890/) (2024)
- [Likelihood-Free Variational Autoencoders](https://mlsanity.com/paper/2504.17622/) (2025)
- [Exploring bidirectional bounds for minimax-training of Energy-based models](https://mlsanity.com/paper/2506.04609/) (2025)
- [Annealed Langevin Posterior Sampling (ALPS): A Rapid Algorithm for Image Restoration with Multiscale Energy Models](https://mlsanity.com/paper/2601.02594/) (2026)

## Abstract (arXiv)

This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM). The joint training of VAE and latent EBM are based on an objective function that consists of three Kullback-Leibler divergences between three joint distributions on the latent vector and the image, and the objective function is of an elegant symmetric and anti-symmetric form of divergence triangle that seamlessly integrates variational and adversarial learning. In this joint training scheme, the latent EBM serves as a critic of the generator model, while the generator model and the inference model in VAE serve as the approximate synthesis sampler and inference sampler of the latent EBM. Our experiments show that the joint training greatly improves the synthesis quality of the VAE. It also enables learning of an energy function that is capable of detecting out of sample examples for anomaly detection.