mlsanityData to 7 Oct 2026

Learning Energy-Based Prior Model with Diffusion-Amortized MCMC

arXiv 2310.03218 ↗First posted 5 Oct 2023cs.LG cs.AI stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in the field of generative modeling due to its flexibility in the formulation and strong modeling power of the latent space. However, the common practice of learning latent space EBMs with non-convergent short-run MCMC for prior and posterior sampling is hindering the model from further progress; the degenerate MCMC sampling quality in practice often leads to degraded generation quality and instability in training, especially with highly multi-modal and/or high-dimensional target distributions. To remedy this sampling issue, in this paper we introduce a simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based on it. We provide theoretical evidence that the learned amortization of MCMC is a valid long-run MCMC sampler. Experiments on several image modeling benchmark datasets demonstrate the superior performance of our method compared with strong counterparts

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 62 of its 82 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

62 of 82 references.

References · 62

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2013Auto-Encoding Variational Bayescited by 3,980
38 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  6. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  8. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  9. 2021Variational Diffusion Modelscited by 1,145
  10. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  12. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  13. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  14. 2021Score-based Generative Modeling in Latent Spacecited by 634
  15. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  16. 2015Generating Sentences from a Continuous Spacecited by 607
  17. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  18. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  19. 2021GAN Inversion: A Surveycited by 366
  20. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  21. 2017VAE with a VampPriorcited by 247
  22. 2019Diagnosing and Enhancing VAE Modelscited by 192
  23. 2016A Theory of Generative ConvNetcited by 174
  24. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 147
  25. 2019From Variational to Deterministic Autoencoderscited by 131
  26. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  27. 2018Efficient GAN-Based Anomaly Detectioncited by 122
  28. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  29. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  30. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  31. 2020Learning Latent Space Energy-Based Prior Modelcited by 104
  32. 2015Hierarchical Variational Modelscited by 89
  33. 2019Preventing Posterior Collapse with delta-VAEscited by 86
  34. 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
  35. 2022Latent Diffusion Energy-Based Model for Interpretable Text Modelingcited by 67
  36. 2020A Contrastive Learning Approach for Training Variational Autoencoder Priorscited by 65
  37. 2021Controllable and Compositional Generation with Latent-Space Energy-Based Modelscited by 64
  38. 2021Trajectory Prediction with Latent Belief Energy-Based Modelcited by 60

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 11; 2025: 2.

13 papers.

Sort:

By area:

  1. 2024Flow Priors for Linear Inverse Problems via Iterative Corrupted Trajectory Matchingcited by 50
  2. 2024Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Modelscited by 18
  3. 2024DISCO: Efficient Diffusion Solver for Large-Scale Combinatorial Optimization Problemscited by 10
  4. 2024"Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihoodcited by 8
  5. 2024Improving Adversarial Energy-Based Model via Diffusion Processcited by 7
  6. 2024Learning Latent Space Hierarchical EBM Diffusion Modelscited by 6
  7. 2024Watermarking Generative Tabular Datacited by 6
  8. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Modelscited by 4
  9. 2024Latent Space Energy-based Neural ODEscited by 4
  10. 2025Training Latent Diffusion Models with Interacting Particle Algorithmscited by 4
  11. 2025Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamicscited by 3
  12. 2024Variational Potential Flow: A Novel Probabilistic Framework for Energy-Based Generative Modellingcited by 0
1 more
  1. 2024Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximizationcited by 0