mlsanityData to 14 Oct 2026

Diffusion Alignment as Variational Expectation-Maximization

arXiv 2510.00502 ↗First posted 1 Oct 2025cs.LGcited by 4

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion alignment aims to optimize diffusion models for the downstream objective. While existing methods based on reinforcement learning or direct backpropagation achieve considerable success in maximizing rewards, they often suffer from reward over-optimization and mode collapse. We introduce Diffusion Alignment as Variational Expectation-Maximization (DAV), a framework that formulates diffusion alignment as an iterative process alternating between two complementary phases: the E-step and the M-step. In the E-step, we employ test-time search to generate diverse and reward-aligned samples. In the M-step, we refine the diffusion model using samples discovered by the E-step. We demonstrate that DAV can optimize reward while preserving diversity for both continuous and discrete tasks: text-to-image synthesis and DNA sequence design. Our code is available at https://github.com/Jaewoopudding/dav.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 64 of its 85 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

64 of 85 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models25 Jun
  2. 2025Inference-time Scaling of Diffusion Models through Classical Search29 May
  3. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing25 Mar
  4. 2025Dynamic Search for Inference-Time Alignment in Diffusion Models3 Mar
  5. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps16 Jan
  6. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Models12 Jan
  7. 2025Test-time Alignment of Diffusion Models without Reward Over-optimization10 Jan
  8. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding15 Aug
  9. 2024Amortizing intractable inference in diffusion models for vision, language, and control31 May
  10. 2024Gradient Guidance for Diffusion Models: An Optimization Perspective23 Apr
  11. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model17 Mar
  12. 2023Consistency Models2 Mar
8 more
  1. 2023Universal Guidance for Diffusion Models14 Feb
  2. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuning31 Jan
  3. 2022Classifier-Free Diffusion Guidance26 Jul
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  6. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  9. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  10. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  11. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
32 more
  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  2. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  5. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  6. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  9. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  10. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  12. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  13. 2018Maximum a Posteriori Policy Optimisationcited by 328
  14. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  15. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  16. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 143
  17. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  18. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 129
  19. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 101
  20. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  21. 2021Oops I Took A Gradient: Scalable Sampling for Discrete Distributionscited by 92
  22. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 88
  23. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 79
  24. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  25. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  26. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 32
  27. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 30
  28. 2023Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequencescited by 23
  29. 2025Split Gibbs Discrete Diffusion Posterior Samplingcited by 18
  30. 2025Offline Model-Based Optimization: Comprehensive Reviewcited by 15
  31. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  32. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  2. 2026FestDPO: Few-step Generator Alignment with Direct Preference Optimizationcited by 0
  3. 2026MEND: RL For Flow Models via Proximal Velocity Matchingcited by 0
  4. 2026Backpropagating Verifier Rewards Through Continuous Flow Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
  2. 2025Direct Distributional Optimization for Provable Alignment of Diffusion Models
  3. 2026The tractability landscape of diffusion alignment: regularization, rewards, and computational primitives
  4. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser
  5. 2025Inference-Time Alignment of Diffusion Models via Evolutionary Algorithms
  6. 2026Risk-Averse Inference-Time Diffusion Model Alignment with Finite-Sample Guarantees
  7. 2026Best-of-N Guidance for Test-time Diffusion Alignment
  8. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review