# Bi-level Doubly Variational Learning for Energy-based Latent Variable Models

arXiv: https://arxiv.org/abs/2203.14702 · HTML: https://mlsanity.com/paper/2203.14702/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Unrolled Generative Adversarial Networks](https://mlsanity.com/paper/1611.02163/) (2016)
- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Learning Latent Space Energy-Based Prior Model](https://mlsanity.com/paper/2006.08205/) (2020)
- [VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models](https://mlsanity.com/paper/2010.00654/) (2020)
- [A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singleton](https://mlsanity.com/paper/2006.04045/) (2020)
- [Flow Contrastive Estimation of Energy-Based Models](https://mlsanity.com/paper/1912.00589/) (2019)
- [Generalized Energy Based Models](https://mlsanity.com/paper/2003.05033/) (2020)
- [Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling](https://mlsanity.com/paper/2003.06060/) (2020)
- [No MCMC for me: Amortized sampling for fast and stable training of energy-based models](https://mlsanity.com/paper/2010.04230/) (2020)
- [Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler](https://mlsanity.com/paper/2012.14936/) (2020)
- [Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model](https://mlsanity.com/paper/1812.10907/) (2018)
- [ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching](https://mlsanity.com/paper/1709.01215/) (2017)
- [Joint Training of Variational Auto-Encoder and Latent Energy-Based Model](https://mlsanity.com/paper/2006.06059/) (2020)
- [Training Deep Energy-Based Models with f-Divergence Minimization](https://mlsanity.com/paper/2003.03463/) (2020)
- [Learning Energy-Based Models in High-Dimensional Spaces with Multi-scale Denoising Score Matching](https://mlsanity.com/paper/1910.07762/) (2019)
- [Symmetric Variational Autoencoder and Connections to Adversarial Learning](https://mlsanity.com/paper/1709.01846/) (2017)
- [Bi-level Score Matching for Learning Energy-based Latent Variable Models](https://mlsanity.com/paper/2010.07856/) (2020)
- [Bounds all around: training energy-based models with bidirectional bounds](https://mlsanity.com/paper/2111.00929/) (2021)
- [Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models](https://mlsanity.com/paper/2010.08258/) (2020)
- [Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator](https://mlsanity.com/paper/2004.01704/) (2020)

## Built on by (most cited first, 6 of 6)

- [Guiding Energy-based Models via Contrastive Latent Variables](https://mlsanity.com/paper/2303.03023/) (2023)
- [EGC: Image Generation and Classification via a Diffusion Energy-Based Model](https://mlsanity.com/paper/2304.02012/) (2023)
- [Improving Adversarial Energy-Based Model via Diffusion Process](https://mlsanity.com/paper/2403.01666/) (2024)
- [How to Construct Energy for Images? Denoising Autoencoder Can Be Energy Based Model](https://mlsanity.com/paper/2303.03887/) (2023)
- [Exploring bidirectional bounds for minimax-training of Energy-based models](https://mlsanity.com/paper/2506.04609/) (2025)
- [From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation](https://mlsanity.com/paper/2510.07624/) (2025)

## Abstract (arXiv)

Energy-based latent variable models (EBLVMs) are more expressive than conventional energy-based models. However, its potential on visual tasks are limited by its training process based on maximum likelihood estimate that requires sampling from two intractable distributions. In this paper, we propose Bi-level doubly variational learning (BiDVL), which is based on a new bi-level optimization framework and two tractable variational distributions to facilitate learning EBLVMs. Particularly, we lead a decoupled EBLVM consisting of a marginal energy-based distribution and a structural posterior to handle the difficulties when learning deep EBLVMs on images. By choosing a symmetric KL divergence in the lower level of our framework, a compact BiDVL for visual tasks can be obtained. Our model achieves impressive image generation performance over related works. It also demonstrates the significant capacity of testing image reconstruction and out-of-distribution detection.