mlsanityData to 7 Oct 2026

Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser

arXiv 2602.12229 ↗First posted 12 Feb 2026cs.LGcited by 1

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion alignment adapts pretrained diffusion models to sample from reward-tilted distributions along the denoising trajectory. This process naturally admits a Sequential Monte Carlo (SMC) interpretation, where the denoising model acts as a proposal and reward guidance induces importance weights. Motivated by this view, we introduce Variance Minimisation Policy Optimisation (VMPO), which formulates diffusion alignment as minimising the variance of log importance weights rather than directly optimising a Kullback-Leibler (KL) based objective. We prove that the variance objective is minimised by the reward-tilted target distribution and that, under on-policy sampling, its gradient coincides with that of standard KL-based alignment. This perspective offers a common lens for understanding diffusion alignment. Under different choices of potential functions and variance minimisation strategies, VMPO recovers various existing methods, while also suggesting new design directions beyond KL.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 81 of its 97 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

81 of 97 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Process19 Sep
  2. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE29 Jul
  3. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  5. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets10 Dec
  6. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  7. 2024Improving GFlowNets for Text-to-Image Diffusion Alignment2 Jun
  8. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  9. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  10. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  11. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  12. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
2 more
  1. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  2. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 67

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  13. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  14. 2016Variational Inference: A Review for Statisticianscited by 1,466
  15. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  16. 2021Variational Diffusion Modelscited by 1,145
  17. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  18. 2025Qwen-Image Technical Reportcited by 973
  19. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  20. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  21. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  22. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  23. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  24. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  25. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  26. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  27. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  28. 2013Black Box Variational Inferencecited by 317
  29. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  30. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  31. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 218
  32. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 211
  33. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  34. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  35. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 139
  36. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  37. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  38. 2023Better Training of GFlowNets with Local Credit and Incomplete Trajectoriescited by 83

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-time Alignment of Diffusion Models without Reward Over-optimization
  2. 2026Step-level Denoising-time Diffusion Alignment with Multiple Objectives
  3. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
  4. 2026The tractability landscape of diffusion alignment: regularization, rewards, and computational primitives
  5. 2025Diffusion Alignment as Variational Expectation-Maximization
  6. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models
  7. 2025Unified Control for Inference-Time Guidance of Denoising Diffusion Models
  8. 2026iADD: Improving Alignment and Diversity in Diffusion Policy Optimization