mlsanityData to 14 Oct 2026

Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning

arXiv 2502.01819 ↗First posted 3 Feb 2025cs.LG cs.AI math.OCcited by 33

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works in this area use a discrete-time formulation, which is prone to induced discretization errors, and often not applicable to models with higher-order/black-box solvers. The objective of this study is to develop a disciplined approach to fine-tune diffusion models using continuous-time RL, formulated as a stochastic control problem with a reward function that aligns the end result (terminal state) with input prompt. The key idea is to treat score matching as controls or actions, and thereby making connections to policy optimization and regularization in continuous-time RL. To carry out this idea, we lay out a new policy optimization framework for continuous-time RL, and illustrate its potential in enhancing the value networks design space via leveraging the structural property of diffusion models. We validate the advantages of our method by experiments in downstream tasks of fine-tuning large-scale Text2Image models of Stable Diffusion v1.5.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 54 of its 65 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

54 of 65 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  2. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning12 Sep
  3. 2024Reward-Directed Score-Based Diffusion Models via q-Learning7 Sep
  4. 2024Diffusion Policy Policy Optimization1 Sep
  5. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models30 Jun
  6. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  7. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  8. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  9. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  10. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  11. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  12. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
30 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2023Consistency Modelscited by 1,822
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  10. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  11. 2021Variational Diffusion Modelscited by 1,153
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  13. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  14. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 521
  15. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 516
  16. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 445
  17. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  18. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 395
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  20. 2022gDDIM: Generalized denoising diffusion implicit modelscited by 125
  21. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  22. 2022q-Learning in Continuous Timecited by 91
  23. 2020Improving Image Captioning with Better Use of Captionscited by 84
  24. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  25. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  26. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorialcited by 60
  27. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  28. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
  29. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 30
  30. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 18.

33 papers.

Sort:

By area:

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  2. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  3. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matchingcited by 56
  4. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 19
  5. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  6. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  7. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generationcited by 12
  8. 2025HardFlow: Hard-Constrained Sampling for Flow-Matching Models via Trajectory Optimizationcited by 11
  9. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 11
  10. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 10
  11. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  12. 2025PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modelingcited by 7
21 more
  1. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  2. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 5
  3. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  4. 2025MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignmentcited by 4
  5. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 4
  6. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 4
  7. 2026Efficient Adjoint Matching for Fine-tuning Diffusion Modelscited by 4
  8. 2025Is RL fine-tuning harder than regression? A PDE learning approach for diffusion modelscited by 3
  9. 2025Diffusion Generative Models Meet Compressed Sensing, with Applications to Imaging and Financecited by 3
  10. 2025Parallel Diffusion Solver via Residual Dirichlet Policy Optimizationcited by 3
  11. 2026Constrained Flow Optimization via Sequential Fine Tuning for Molecular Designcited by 3
  12. 2025What Happens Next? Next Scene Prediction with a Unified Video Modelcited by 2
  13. 2026Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewardscited by 2
  14. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 2
  15. 2026Amortized Guidance for Image Inpainting with Pretrained Diffusion Modelscited by 1
  16. 2026DriveAnchor: Progressive Anchor-based Flow Learning for Autonomous Driving Planningcited by 1
  17. 2026DOHF: Online Diffusion Fine-tuning with Doob's h-transform Guidancecited by 1
  18. 2026Explicit Critic Guidance for Aligning Diffusion Modelscited by 0
  19. 2026ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learningcited by 0
  20. 2026LeanGRPO: Eliminating Redundant Recomputation in Diffusion RLcited by 0
  21. 2026Balancing Emotional Alignment and Semantic Consistency in Image Generation via Reinforcement Learning with Valence-Arousal Anchoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  2. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence
  3. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
  4. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review
  5. 2026Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
  6. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods
  7. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
  8. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL