# Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood

arXiv: https://arxiv.org/abs/2309.05153 · HTML: https://mlsanity.com/paper/2309.05153/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 74 of its 105 references that are in the map.
- First posted on arXiv 10 Sep 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 74)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)

## Built on by (most cited first, 19 of 19)

- [Energy Matching: Unifying Flow Matching and Energy-Based Models for Generative Modeling](https://mlsanity.com/paper/2504.10612/) (2025)
- [Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transport](https://mlsanity.com/paper/2402.05443/) (2024)
- [Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo](https://mlsanity.com/paper/2502.12786/) (2025)
- [Learning Energy-Based Prior Model with Diffusion-Amortized MCMC](https://mlsanity.com/paper/2310.03218/) (2023)
- [Learned Reference-based Diffusion Sampling for multi-modal distributions](https://mlsanity.com/paper/2410.19449/) (2024)
- [An efficient probabilistic hardware architecture for diffusion-like models](https://mlsanity.com/paper/2510.23972/) (2025)
- [Improving Adversarial Energy-Based Model via Diffusion Process](https://mlsanity.com/paper/2403.01666/) (2024)
- [Learning Latent Space Hierarchical EBM Diffusion Models](https://mlsanity.com/paper/2405.13910/) (2024)
- [Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamics](https://mlsanity.com/paper/2510.12311/) (2025)
- [Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Models](https://mlsanity.com/paper/2601.22057/) (2026)
- [Learning Energy-Based Generative Models via Potential Flow: A Variational Principle Approach to Probability Density Homotopy Matching](https://mlsanity.com/paper/2504.16262/) (2025)
- [Dominant vs. Dominated: Concept-Level Generative Collapse in Diffusion Models](https://mlsanity.com/paper/2512.20666/) (2025)
- [Variational Potential Flow: A Novel Probabilistic Framework for Energy-Based Generative Modelling](https://mlsanity.com/paper/2407.15238/) (2024)
- [Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection](https://mlsanity.com/paper/2409.03801/) (2024)
- [Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization](https://mlsanity.com/paper/2410.02628/) (2024)
- [Exploring bidirectional bounds for minimax-training of Energy-based models](https://mlsanity.com/paper/2506.04609/) (2025)
- [Distance Marching for Generative Modeling](https://mlsanity.com/paper/2602.02928/) (2026)
- [From Noise to Control: Parameterized Diffusion Policies](https://mlsanity.com/paper/2606.00336/) (2026)
- [Comparing SGLD and a fixed-noise predictor-corrector adaptation in canonical joint energy-based models on CIFAR-10](https://mlsanity.com/paper/2608.05025/) (2026)

## Abstract (arXiv)

Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming, and there exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models. To close this gap, inspired by the recent efforts of learning EBMs by maximizing diffusion recovery likelihood (DRL), we propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs defined on increasingly noisy versions of a dataset, paired with an initializer model for each EBM. At each noise level, the two models are jointly estimated within a cooperative training framework: samples from the initializer serve as starting points that are refined by a few MCMC sampling steps from the EBM. The EBM is then optimized by maximizing recovery likelihood, while the initializer model is optimized by learning from the difference between the refined samples and the initial samples. In addition, we made several practical designs for EBM training to further improve the sample quality. Combining these advances, our approach significantly boost the generation performance compared to existing EBM methods on CIFAR-10 and ImageNet datasets. We also demonstrate the effectiveness of our models for several downstream tasks, including classifier-free guided generation, compositional generation, image inpainting and out-of-distribution detection.