# Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo

arXiv: https://arxiv.org/abs/2502.12786 · HTML: https://mlsanity.com/paper/2502.12786/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 34 of its 96 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 34)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Fast Sampling of Diffusion Models with Exponential Integrator](https://mlsanity.com/paper/2204.13902/) (2022)
- [Guiding a Diffusion Model with a Bad Version of Itself](https://mlsanity.com/paper/2406.02507/) (2024)
- [Rectified Flow: A Marginal Preserving Approach to Optimal Transport](https://mlsanity.com/paper/2209.14577/) (2022)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [A General Framework for Inference-time Scaling and Steering of Diffusion Models](https://mlsanity.com/paper/2501.06848/) (2025)
- [I2SB: Image-to-Image Schr"odinger Bridge](https://mlsanity.com/paper/2302.05872/) (2023)
- [Diffusion Schr"odinger Bridge Matching](https://mlsanity.com/paper/2303.16852/) (2023)
- [Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding](https://mlsanity.com/paper/2408.08252/) (2024)
- [Learning Energy-Based Models by Diffusion Recovery Likelihood](https://mlsanity.com/paper/2012.08125/) (2020)
- [Improving the Training of Rectified Flows](https://mlsanity.com/paper/2405.20320/) (2024)
- [TFG: Unified Training-Free Guidance for Diffusion Models](https://mlsanity.com/paper/2409.15761/) (2024)
- [Deconstructing Denoising Diffusion Models for Self-Supervised Learning](https://mlsanity.com/paper/2401.14404/) (2024)
- [Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo](https://mlsanity.com/paper/2404.17546/) (2024)
- [Energy-Based Diffusion Language Models for Text Generation](https://mlsanity.com/paper/2410.21357/) (2024)
- [Classifier-Free Guidance is a Predictor-Corrector](https://mlsanity.com/paper/2408.09000/) (2024)
- [Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs](https://mlsanity.com/paper/2306.03081/) (2023)
- [Particle Denoising Diffusion Sampler](https://mlsanity.com/paper/2402.06320/) (2024)
- [DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised h-transform](https://mlsanity.com/paper/2406.01781/) (2024)
- [Blindness of score-based methods to isolated components and mixing proportions](https://mlsanity.com/paper/2008.10087/) (2020)
- [Simple ReFlow: Improved Techniques for Fast Flow Models](https://mlsanity.com/paper/2410.07815/) (2024)
- [Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlo](https://mlsanity.com/paper/2502.06079/) (2025)
- [Dynamic Negative Guidance of Diffusion Models](https://mlsanity.com/paper/2410.14398/) (2024)
- [On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models](https://mlsanity.com/paper/2402.03845/) (2024)
- [Towards Differentiable Resampling](https://mlsanity.com/paper/2004.11938/) (2020)
- [Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood](https://mlsanity.com/paper/2309.05153/) (2023)

## Built on by (most cited first, 20 of 20)

- [Generative Trajectory Stitching through Diffusion Composition](https://mlsanity.com/paper/2503.05153/) (2025)
- [Progressive Inference-Time Annealing of Diffusion Models for Sampling from Boltzmann Densities](https://mlsanity.com/paper/2506.16471/) (2025)
- [DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Models](https://mlsanity.com/paper/2509.21655/) (2025)
- [Learning normalized image densities via dual score matching](https://mlsanity.com/paper/2506.05310/) (2025)
- [RNE: plug-and-play diffusion inference-time control and energy-based training](https://mlsanity.com/paper/2506.05668/) (2025)
- [CREPE: Controlling Diffusion with Replica Exchange](https://mlsanity.com/paper/2509.23265/) (2025)
- [Product of Experts for Visual Generation](https://mlsanity.com/paper/2506.08894/) (2025)
- [PriorGuide: Test-Time Prior Adaptation for Simulation-Based Inference](https://mlsanity.com/paper/2510.13763/) (2025)
- [Compositional Visual Planning via Inference-Time Diffusion Scaling](https://mlsanity.com/paper/2603.02646/) (2026)
- [Learning Energy-Based Models from Stochastic Interpolants using Spatiotemporal Differences](https://mlsanity.com/paper/2605.26850/) (2026)
- [Ctrl-VI: Controllable Video Synthesis via Variational Inference](https://mlsanity.com/paper/2510.07670/) (2025)
- [CMAD: Cooperative Multi-Agent Diffusion via Stochastic Optimal Control](https://mlsanity.com/paper/2602.10933/) (2026)
- [Energy Guided Geometric Flow Matching](https://mlsanity.com/paper/2509.25230/) (2025)
- [Annealed Langevin Posterior Sampling (ALPS): A Rapid Algorithm for Image Restoration with Multiscale Energy Models](https://mlsanity.com/paper/2601.02594/) (2026)
- [Seasoning Generative Models for a Generalization Aftertaste](https://mlsanity.com/paper/2603.18817/) (2026)
- [Learning Normalized Energy Models for Linear Inverse Problems](https://mlsanity.com/paper/2605.15487/) (2026)
- [Comparing SGLD and a fixed-noise predictor-corrector adaptation in canonical joint energy-based models on CIFAR-10](https://mlsanity.com/paper/2608.05025/) (2026)
- [Error-Corrected Inference-Time Scaling for Imperfect Diffusion Models](https://mlsanity.com/paper/2610.01933/) (2026)
- [Risk-Calibrated Proposal Transport for Finite-Particle Diffusion Steering](https://mlsanity.com/paper/2610.04171/) (2026)
- [Steering Diffusion Models to Rare Events with Sequential Monte Carlo](https://mlsanity.com/paper/2610.08652/) (2026)

## Abstract (arXiv)

Diffusion models may be formulated as a time-indexed sequence of energy-based models, where the score corresponds to the negative gradient of an energy function. As opposed to learning the score directly, an energy parameterization is attractive as the energy itself can be used to control generation via Monte Carlo samplers. Architectural constraints and training instability in energy parameterized models have so far yielded inferior performance compared to directly approximating the score or denoiser. We address these deficiencies by introducing a novel training regime for the energy function through distillation of pre-trained diffusion models, resembling a Helmholtz decomposition of the score vector field. We further showcase the synergies between energy and score by casting the diffusion sampling procedure as a Feynman Kac model where sampling is controlled using potentials from the learnt energy functions. The Feynman Kac model formalism enables composition and low temperature sampling through sequential Monte Carlo.