mlsanityData to 7 Oct 2026

Training Diffusion Models with Reinforcement Learning

arXiv 2305.13301 ↗First posted 22 May 2023cs.LG cs.AI cs.CVcited by 939

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion models are a class of flexible generative models trained with an approximation to the log-likelihood objective. However, most use cases of diffusion models are not concerned with likelihoods, but instead with downstream objectives such as human-perceived image quality or drug effectiveness. In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for such objectives. We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms, which we refer to as denoising diffusion policy optimization (DDPO), that are more effective than alternative reward-weighted likelihood approaches. Empirically, DDPO is able to adapt text-to-image diffusion models to objectives that are difficult to express via prompting, such as image compressibility, and those derived from human feedback, such as aesthetic quality. Finally, we show that DDPO can improve prompt-image alignment using feedback from a vision-language model without the need for additional data collection or human annotation. The project's website can be found at http://rl-diffusion.github.io .

Key facts. Cited by 939 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 52 of its 69 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

52 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
36 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  15. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  16. 2021Variational Diffusion Modelscited by 1,145
  17. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  18. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  19. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  20. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  21. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  22. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  23. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  24. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  25. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  26. 2023Universal Guidance for Diffusion Modelscited by 426
  27. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  28. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  29. 2022Teaching language models to support answers with verified quotescited by 290
  30. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  31. 2021Crystal Diffusion Variational Autoencoder for Periodic Material Generationcited by 268
  32. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  33. 2022Structure-based Drug Design with Equivariant Diffusion Modelscited by 213
  34. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  35. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  36. 2015Data Generation as Sequential Decision Makingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 177; 2025: 334; 2026: 386.

939 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  3. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  6. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  7. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 388
  8. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  9. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  10. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  11. 2025Improving Video Generation with Human Feedbackcited by 267
  12. 2024Diffusion Policy Policy Optimizationcited by 255
38 more
  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  5. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  6. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  7. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  8. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  9. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  10. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  11. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  12. 2023Video Language Planningcited by 160
  13. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  14. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  15. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  16. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  17. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  18. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  19. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  20. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  21. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  22. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  23. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  24. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  25. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
  26. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  27. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  28. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  29. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  30. 2025Flow Matching Policy Gradientscited by 86
  31. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  32. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  33. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  34. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  35. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  36. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  37. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
  38. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77

50 most cited of 939 All 939 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large-scale Reinforcement Learning for Diffusion Models
  2. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
  3. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  4. 2026Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusion
  5. 2025D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples
  6. 2026Learning Sampling Parameters for Diffusion Models
  7. 2026On-Policy Self-Distillation in Diffusion Models
  8. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models