mlsanityData to 14 Oct 2026

Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review

arXiv 2407.13734 ↗First posted 18 Jul 2024cs.LG cs.AI q-bio.QMcited by 98

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. While diffusion models are widely known to provide excellent generative modeling capability, practical applications in domains such as biology require generating samples that maximize some desired metric (e.g., translation efficiency in RNA, docking score in molecules, stability in protein). In these cases, the diffusion model can be optimized not only to generate realistic samples but also to explicitly maximize the measure of interest. Such methods are based on concepts from reinforcement learning (RL). We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning, tailored specifically for fine-tuning diffusion models. We aim to explore fundamental aspects such as the strengths and limitations of different RL-based fine-tuning algorithms across various scenarios, the benefits of RL-based fine-tuning compared to non-RL-based approaches, and the formal objectives of RL-based fine-tuning (target distributions). Additionally, we aim to examine their connections with related topics such as classifier guidance, Gflownets, flow-based diffusion models, path integral control theory, and sampling from unnormalized distributions such as MCMC. The code of this tutorial is available at https://github.com/masa-ue/RLfinetuning_Diffusion_Bioseq

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 87 of its 110 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

87 of 110 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learning17 Jun
  2. 2024Feedback Efficient Online Fine-Tuning of Diffusion Models26 Feb
  3. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control23 Feb
  4. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  5. 2023Diffusion Models for Reinforcement Learning: A Survey2 Nov
  6. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  7. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  8. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvement13 Jul
  9. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  10. 2023Training Diffusion Models with Reinforcement Learning22 May
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2023Towards Controllable Diffusion Models via Reward-Guided Exploration14 Apr
2 more
  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 73

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022Flow Matching for Generative Modelingcited by 5,702
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2018Neural Ordinary Differential Equationscited by 3,779
38 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2022Video Diffusion Modelscited by 2,305
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  7. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  13. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
  14. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  15. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  16. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  17. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  18. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  19. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 516
  20. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 508
  21. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  22. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  23. 2023Universal Guidance for Diffusion Modelscited by 437
  24. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  25. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  26. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 305
  27. 2021GFlowNet Foundationscited by 290
  28. 2019A Theory of Regularized Markov Decision Processescited by 286
  29. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 283
  30. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  31. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  32. 2022A Survey on Generative Diffusion Modelcited by 240
  33. 2022Riemannian Score-Based Generative Modellingcited by 219
  34. 2018Optimization of Molecules via Deep Reinforcement Learningcited by 200
  35. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  36. 2017A unified view of entropy-regularized Markov decision processescited by 182
  37. 2021Path Integral Sampler: a stochastic control approach for samplingcited by 181
  38. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 38; 2026: 44.

98 papers.

Sort:

By area:

  1. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  2. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 136
  3. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 96
  4. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 88
  5. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 79
  6. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  7. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 42
  8. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  9. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  10. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 38
  11. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 37
  12. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
38 more
  1. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 32
  2. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 24
  3. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  4. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  5. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 19
  6. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  7. 2024Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewardcited by 16
  8. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  9. 2024Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximizationcited by 14
  10. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  11. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 14
  12. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 11
  13. 2026Discrete Adjoint Matchingcited by 11
  14. 2026Training-Free Adaptation of Diffusion Models via Doob's h-Transformcited by 11
  15. 2025Revisiting Non-Acyclic GFlowNets in Discrete Environmentscited by 10
  16. 2025Composition and Alignment of Diffusion Models using Constrained Learningcited by 10
  17. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
  18. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 9
  19. 2025Towards a unified framework for guided diffusion modelscited by 8
  20. 2025Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generationcited by 7
  21. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 7
  22. 2025Towards Better Optimization For Listwise Preference in Diffusion Modelscited by 7
  23. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 7
  24. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  25. 2024Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigationcited by 6
  26. 2025Adaptive Destruction Processes for Diffusion Samplerscited by 6
  27. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 6
  28. 2025Reinforced sequential Monte Carlo for amortised samplingcited by 6
  29. 2026How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidancecited by 6
  30. 2026Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?cited by 6
  31. 2025Direct Distributional Optimization for Provable Alignment of Diffusion Modelscited by 5
  32. 2025Relative Trajectory Balance is equivalent to Trust-PCLcited by 5
  33. 2025Diffusion Fine-Tuning via Reparameterized Policy Gradient of the Soft Q-Functioncited by 5
  34. 2026Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generationcited by 5
  35. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 5
  36. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 5
  37. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 5
  38. 2026Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusioncited by 5

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design
  2. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  3. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
  4. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL
  5. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning
  6. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
  7. 2026Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
  8. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models