mlsanityData to 14 Oct 2026

Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood

arXiv 2309.05153 ↗First posted 10 Sep 2023stat.ML cs.LGcited by 19

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming, and there exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models. To close this gap, inspired by the recent efforts of learning EBMs by maximizing diffusion recovery likelihood (DRL), we propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs defined on increasingly noisy versions of a dataset, paired with an initializer model for each EBM. At each noise level, the two models are jointly estimated within a cooperative training framework: samples from the initializer serve as starting points that are refined by a few MCMC sampling steps from the EBM. The EBM is then optimized by maximizing recovery likelihood, while the initializer model is optimized by learning from the difference between the refined samples and the initial samples. In addition, we made several practical designs for EBM training to further improve the sample quality. Combining these advances, our approach significantly boost the generation performance compared to existing EBM methods on CIFAR-10 and ImageNet datasets. We also demonstrate the effectiveness of our models for several downstream tasks, including classifier-free guided generation, compositional generation, image inpainting and out-of-distribution detection.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 74 of its 105 references that are in the map. First posted on arXiv 10 Sep 2023.

Builds on

74 of 105 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC22 Feb
  2. 2022Classifier-Free Diffusion Guidance26 Jul
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  5. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  7. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANs15 Dec
  8. 2021Variational Diffusion Models1 Jul
  9. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation10 Jun
  10. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  11. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  12. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
4 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Implicit Models6 Oct
  3. 2020Denoising Diffusion Probabilistic Models19 Jun
  4. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 58

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2016Improved Techniques for Training GANscited by 2,970
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
38 more
  1. 2017Improved Training of Wasserstein GANscited by 2,235
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  3. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  4. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  5. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  7. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  8. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  9. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 363
  10. 2019Residual Flows for Invertible Generative Modelingcited by 257
  11. 2016A Theory of Generative ConvNetcited by 174
  12. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 148
  13. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 130
  14. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 123
  15. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  16. 2020Learning Latent Space Energy-Based Prior Modelcited by 104
  17. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 104
  18. 2021GraphEBM: Molecular Graph Generation with Energy-Based Modelscited by 85
  19. 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
  20. 2020Generalized Energy Based Modelscited by 73
  21. 2020Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Samplingcited by 71
  22. 2020No MCMC for me: Amortized sampling for fast and stable training of energy-based modelscited by 64
  23. 2016Cooperative Training of Descriptor and Generator Networkscited by 60
  24. 2020Dual Contradistinctive Generative Autoencodercited by 58
  25. 2020Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classificationcited by 57
  26. 2018Learning Descriptor Networks for 3D Shape Synthesis and Analysiscited by 56
  27. 2019Maximum Entropy Generators for Energy-Based Modelscited by 56
  28. 2022A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Modelcited by 44
  29. 2020Learning Energy-Based Model with Variational Auto-Encoder as Amortized Samplercited by 43
  30. 2016Deep Directed Generative Models with Energy-Based Probability Estimationcited by 34
  31. 2018Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Modelcited by 34
  32. 2017Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Samplingcited by 32
  33. 2018Autoregressive Quantile Networks for Generative Modelingcited by 31
  34. 2020Joint Training of Variational Auto-Encoder and Latent Energy-Based Modelcited by 30
  35. 2019Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patternscited by 22
  36. 2022End-to-End Stochastic Optimization with Energy-Based Modelcited by 19
  37. 2017Wasserstein Introspective Neural Networkscited by 18
  38. 2022Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Modelcited by 18

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 7; 2026: 4.

19 papers.

Sort:

By area:

  1. 2025Energy Matching: Unifying Flow Matching and Energy-Based Models for Generative Modelingcited by 27
  2. 2024Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transportcited by 23
  3. 2025Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlocited by 20
  4. 2023Learning Energy-Based Prior Model with Diffusion-Amortized MCMCcited by 13
  5. 2024Learned Reference-based Diffusion Sampling for multi-modal distributionscited by 9
  6. 2025An efficient probabilistic hardware architecture for diffusion-like modelscited by 8
  7. 2024Improving Adversarial Energy-Based Model via Diffusion Processcited by 7
  8. 2024Learning Latent Space Hierarchical EBM Diffusion Modelscited by 6
  9. 2025Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamicscited by 5
  10. 2026Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Modelscited by 3
  11. 2025Learning Energy-Based Generative Models via Potential Flow: A Variational Principle Approach to Probability Density Homotopy Matchingcited by 2
  12. 2025Dominant vs. Dominated: Concept-Level Generative Collapse in Diffusion Modelscited by 1
7 more
  1. 2024Variational Potential Flow: A Novel Probabilistic Framework for Energy-Based Generative Modellingcited by 0
  2. 2024Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detectioncited by 0
  3. 2024Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximizationcited by 0
  4. 2025Exploring bidirectional bounds for minimax-training of Energy-based modelscited by 0
  5. 2026Distance Marching for Generative Modelingcited by 0
  6. 2026From Noise to Control: Parameterized Diffusion Policiescited by 0
  7. 2026Comparing SGLD and a fixed-noise predictor-corrector adaptation in canonical joint energy-based models on CIFAR-10cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Diffusive Classification Loss for Learning Energy-based Generative Models
  2. 2025Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
  3. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models
  4. 2022Progressive Distillation for Fast Sampling of Diffusion Models
  5. 2024Energy based diffusion generator for efficient sampling of Boltzmann distributions
  6. 2025Diffusion Models: A Mathematical Introduction
  7. 2025Adaptive Destruction Processes for Diffusion Samplers
  8. 2023Explaining the effects of non-convergent sampling in the training of Energy-Based Models