mlsanityData to 14 Oct 2026

Explicit Critic Guidance for Aligning Diffusion Models

arXiv 2605.27736 ↗First posted 26 May 2026cs.LG cs.CVcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Online reinforcement learning is becoming increasingly important for aligning diffusion models with non-differentiable objectives. However, existing methods still face limitations in assigning fine-grained credit along denoising trajectories and in realizing stable value-based optimization. We propose a state-aligned latent actor-critic framework for diffusion post-training, in which the diffusion model serves as its own timestep-conditioned value function and predicts values directly on noisy latent states. This enables trajectory-level PPO training, supports stable actor-critic optimization with simple conditioning and value pretraining strategies, and naturally allows the learned critic to be reused for inference-time steering. We further extend the framework to multi-reward optimization, where joint training with complementary rewards helps alleviate reward hacking. Across both UNet- and DiT-based backbones, our method consistently outperforms prior group-relative RL and actor-critic baselines on single-reward and multi-reward benchmarks, while test-time steering provides additional gains in generation quality.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 77 of its 90 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

77 of 90 references.

Closest ancestors · 23

Same direction, newest first

  1. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment28 Jan
  2. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models9 Dec
  3. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models7 Sep
  4. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE29 Jul
  5. 2025Flow Matching Policy Gradients28 Jul
  6. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  7. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  8. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning2 Mar
  9. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  10. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning3 Feb
  11. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization3 Feb
  12. 2024Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimation22 Nov
11 more
  1. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization6 Jun
  2. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
  3. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  4. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preference13 Feb
  5. 2024Large-scale Reinforcement Learning for Diffusion Models20 Jan
  6. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  7. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  8. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  9. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  10. 2023Training Diffusion Models with Reinforcement Learning22 May
  11. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022Flow Matching for Generative Modelingcited by 5,702
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  13. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  14. 2025Qwen-Image Technical Reportcited by 1,044
  15. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  16. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  17. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  18. 2025Group Sequence Policy Optimizationcited by 835
  19. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  20. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  21. 2025Seed1.5-VL Technical Reportcited by 373
  22. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 232
  23. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 224
  24. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
  25. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  26. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  27. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  28. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 151
  29. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  30. 2024Video Diffusion Alignment via Reward Gradientscited by 107
  31. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 82
  32. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  33. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 49
  34. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  35. 2024Scaling Laws For Diffusion Transformerscited by 44
  36. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 41
  37. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  38. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 22

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026iADD: Improving Alignment and Diversity in Diffusion Policy Optimization
  2. 2026DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
  3. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  4. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models
  5. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  6. 2026Latent Reward Registers for Diffusion Preference Alignment
  7. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
  8. 2026ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcement