mlsanityData to 14 Oct 2026

Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning

arXiv 2409.08400 ↗First posted 12 Sep 2024cs.LG cs.AIcited by 19

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

Reinforcement Learning from human feedback (RLHF) has been shown a promising direction for aligning generative models with human intent and has also been explored in recent works for alignment of diffusion generative models. In this work, we provide a rigorous treatment by formulating the task of fine-tuning diffusion models, with reward functions learned from human feedback, as an exploratory continuous-time stochastic control problem. Our key idea lies in treating the score-matching functions as controls/actions, and upon this, we develop a unified framework from a continuous-time perspective, to employ reinforcement learning (RL) algorithms in terms of improving the generation quality of diffusion models. We also develop the corresponding continuous-time RL theory for policy optimization and regularization under assumptions of stochastic different equations driven environment. Experiments on the text-to-image (T2I) generation will be reported in the accompanied paper.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 38 of its 58 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

38 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review18 Jul
  2. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  3. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  4. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
22 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  2. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  3. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  4. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  5. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 521
  6. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 445
  7. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  8. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  9. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 226
  10. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  11. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matchingcited by 183
  12. 2022gDDIM: Generalized denoising diffusion implicit modelscited by 125
  13. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  14. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 97
  15. 2022q-Learning in Continuous Timecited by 91
  16. 2020Improving Image Captioning with Better Use of Captionscited by 84
  17. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  18. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  19. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorialcited by 60
  20. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  21. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
  22. 2023Data-Driven Merton's Strategies via Policy Randomizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  2. 2025RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learningcited by 23
  3. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 19
  4. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 17
  5. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 14
  6. 2024Regret of exploratory policy improvement and q-learningcited by 12
  7. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 11
  8. 2025Composition and Alignment of Diffusion Models using Constrained Learningcited by 10
  9. 2025Towards a unified framework for guided diffusion modelscited by 8
  10. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  11. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 7
  12. 2025Direct Distributional Optimization for Provable Alignment of Diffusion Modelscited by 5
7 more
  1. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 5
  2. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 4
  3. 2026A Unified Density Operator View of Flow Control and Mergingcited by 2
  4. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 2
  5. 2026DOHF: Online Diffusion Fine-tuning with Doob's h-transform Guidancecited by 1
  6. 2026Rethinking Floorplan Generation with Space Syntax: From Design Theory to Model Improvementcited by 0
  7. 2026ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward-Directed Score-Based Diffusion Models via q-Learning
  2. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
  3. 2026Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
  4. 2023Diffusion Models for Reinforcement Learning: A Survey
  5. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  6. 2025Data-regularized Reinforcement Learning for Diffusion Models at Scale
  7. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
  8. 2026TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward