mlsanityData to 7 Oct 2026

DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models

arXiv 2305.16381 ↗First posted 25 May 2023cs.LG cs.CVcited by 431

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Learning from human feedback has been shown to improve text-to-image models. These techniques first learn a reward function that captures what humans care about in the task and then improve the models based on the learned reward function. Even though relatively simple approaches (e.g., rejection sampling based on reward scores) have been investigated, fine-tuning text-to-image models with the reward function remains challenging. In this work, we propose using online reinforcement learning (RL) to fine-tune text-to-image models. We focus on diffusion models, defining the fine-tuning task as an RL problem, and updating the pre-trained text-to-image diffusion models using policy gradient to maximize the feedback-trained reward. Our approach, coined DPOK, integrates policy optimization with KL regularization. We conduct an analysis of KL regularization for both RL fine-tuning and supervised fine-tuning. In our experiments, we show that DPOK is generally superior to supervised fine-tuning with respect to both image-text alignment and image quality. Our code is available at https://github.com/google-research/google-research/tree/master/dpok.

Key facts. Cited by 431 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 43 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

40 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Training Diffusion Models with Reinforcement Learning22 May
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
26 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  13. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  14. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  15. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  16. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  17. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  18. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  19. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  20. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  21. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  22. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  23. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  24. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  25. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  26. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmarkcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 77; 2025: 148; 2026: 181.

431 papers.

Sort:

By area:

  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  3. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  7. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  8. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  9. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  10. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  11. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  12. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
38 more
  1. 2023Rich Human Feedback for Text-to-Image Generationcited by 139
  2. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  3. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  4. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  5. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  6. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  7. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  8. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  9. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  10. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  11. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  12. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  13. 2024Gradient Guidance for Diffusion Models: An Optimization Perspectivecited by 59
  14. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Modelscited by 59
  15. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  16. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 59
  17. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  18. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  19. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  20. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  21. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
  22. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  23. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  24. 2025On the Guidance of Flow Matchingcited by 44
  25. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  26. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 41
  27. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  28. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularizationcited by 40
  29. 2023Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generationcited by 38
  30. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 38
  31. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  32. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  33. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38
  34. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Modelscited by 36
  35. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
  36. 2024Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generationcited by 35
  37. 2024Reward Guided Latent Consistency Distillationcited by 35
  38. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35

50 most cited of 431 All 431 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning
  2. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
  3. 2024Large-scale Reinforcement Learning for Diffusion Models
  4. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation
  5. 2026PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Model
  6. 2024David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training
  7. 2025DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models
  8. 2026Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusion