mlsanityData to 7 Oct 2026

Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC

arXiv 2302.11552 ↗First posted 22 Feb 2023cs.LG cs.AI cs.CVcited by 216

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Since their introduction, diffusion models have quickly become the prevailing approach to generative modeling in many domains. They can be interpreted as learning the gradients of a time-varying sequence of log-probability density functions. This interpretation has motivated classifier-based and classifier-free guidance as methods for post-hoc control of diffusion models. In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance. In particular, we investigate why certain types of composition fail using current techniques and present a number of solutions. We conclude that the sampler (not the model) is responsible for this failure and propose new samplers, inspired by MCMC, which enable successful compositional generation. Further, we propose an energy-based parameterization of diffusion models which enables the use of new compositional operators and more sophisticated, Metropolis-corrected samplers. Intriguingly we find these samplers lead to notable improvements in compositional generation across a wide set of problems such as classifier-guided ImageNet modeling and compositional text-to-image generation.

Key facts. Cited by 216 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 30 of its 52 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

30 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Classifier-Free Diffusion Guidance26 Jul
  2. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations14 Jul
  3. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  11. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  12. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
13 more
  1. 2016A Theory of Generative ConvNetcited by 174
  2. 2016High-dimensional Bayesian inference via the Unadjusted Langevin Algorithmcited by 144
  3. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  4. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  5. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  6. 2021Deep Generative Learning via Schr"odinger Bridgecited by 103
  7. 2021Learning to Compose Visual Relationscited by 72
  8. 2022Equivariant Energy-Guided SDE for Inverse Molecular Designcited by 72
  9. 2020No MCMC for me: Amortized sampling for fast and stable training of energy-based modelscited by 64
  10. 2020Learning Energy-Based Model with Variational Auto-Encoder as Amortized Samplercited by 43
  11. 2021MCMC Variational Inference via Uncorrected Hamiltonian Annealingcited by 39
  12. 2014Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealingcited by 18
  13. 2019Neural Networks with Cheap Differential Operatorscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 68; 2025: 64; 2026: 51.

216 papers.

Sort:

By area:

  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  3. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  4. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 218
  5. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  6. 2024RoboDreamer: Learning Compositional World Models for Robot Imaginationcited by 187
  7. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  8. 2025History-Guided Video Diffusioncited by 165
  9. 2023Video Language Planningcited by 160
  10. 2024CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Modelscited by 138
  11. 2023Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problemscited by 129
  12. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
38 more
  1. 2024TFG: Unified Training-Free Guidance for Diffusion Modelscited by 103
  2. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  3. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 87
  4. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  5. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
  6. 2024Multi-LoRA Composition for Image Generationcited by 80
  7. 2024Classifier-Free Guidance is a Predictor-Correctorcited by 79
  8. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  9. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 70
  10. 2023Direct Diffusion Bridge using Data Consistency for Inverse Problemscited by 65
  11. 2023Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Modelscited by 65
  12. 2025Proteina: Scaling Flow-based Protein Structure Generative Modelscited by 63
  13. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 51
  14. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Modelscited by 51
  15. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
  16. 2023GenPose: Generative Category-level Object Pose Estimation via Diffusion Modelscited by 47
  17. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  18. 2023Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Modelscited by 46
  19. 2024What does guidance do? A fine-grained analysis in a simple settingcited by 45
  20. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  21. 2024Compositional Generative Modeling: A Single Model is Not All You Needcited by 41
  22. 2024Potential Based Diffusion Motion Planningcited by 40
  23. 2023Compositional Diffusion-Based Continuous Constraint Solverscited by 39
  24. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 39
  25. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  26. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  27. 2024Learning Iterative Reasoning through Energy Diffusioncited by 36
  28. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  29. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35
  30. 2024Inference-Time Policy Steering through Human Interactionscited by 35
  31. 2023Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Modelcited by 33
  32. 2024Diffusion Models as Constrained Samplers for Optimization with Unknown Constraintscited by 33
  33. 2026Q-learning with Adjoint Matchingcited by 33
  34. 2023DiffuseBot: Breeding Soft Robots With Physics-Augmented Generative Diffusion Modelscited by 30
  35. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  36. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  37. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  38. 2024Dynamic Negative Guidance of Diffusion Modelscited by 28

50 most cited of 216 All 216 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Composition and Alignment of Diffusion Models using Constrained Learning
  2. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation
  3. 2022On Distillation of Guided Diffusion Models
  4. 2026TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward
  5. 2026Catastrophic Compositional Generation: Why Vanilla Diffusion Models Fail to Extrapolate
  6. 2025Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
  7. 2022Understanding Diffusion Models: A Unified Perspective
  8. 2025Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo