mlsanityData to 7 Oct 2026

Directly Fine-Tuning Diffusion Models on Differentiable Rewards

arXiv 2309.17400 ↗First posted 29 Sep 2023cs.CV cs.LGcited by 431

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

We present Direct Reward Fine-Tuning (DRaFT), a simple and effective method for fine-tuning diffusion models to maximize differentiable reward functions, such as scores from human preference models. We first show that it is possible to backpropagate the reward function gradient through the full sampling procedure, and that doing so achieves strong performance on a variety of rewards, outperforming reinforcement learning-based approaches. We then propose more efficient variants of DRaFT: DRaFT-K, which truncates backpropagation to only the last K steps of sampling, and DRaFT-LV, which obtains lower-variance gradient estimates for the case when K=1. We show that our methods work well for a variety of reward functions and can be used to substantially improve the aesthetic quality of images generated by Stable Diffusion 1.4. Finally, we draw connections between our approach and prior work, providing a unifying perspective on the design space of gradient-based fine-tuning algorithms.

Key facts. Cited by 431 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 61 of its 72 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

61 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  2. 2023Training Diffusion Models with Reinforcement Learning22 May
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2020Learning to summarize from human feedbackcited by 2,562
  13. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  14. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  15. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  17. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  18. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  19. 2021Variational Diffusion Modelscited by 1,145
  20. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  21. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  22. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  23. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  24. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  25. 2021Robust fine-tuning of zero-shot modelscited by 785
  26. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  27. 2022On Distillation of Guided Diffusion Modelscited by 721
  28. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  29. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  30. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  31. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  32. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  33. 2023Universal Guidance for Diffusion Modelscited by 426
  34. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  35. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  36. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  37. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  38. 2021Recursively Summarizing Books with Human Feedbackcited by 287

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 92; 2025: 174; 2026: 155.

431 papers.

Sort:

By area:

  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  3. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  4. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  5. 2025Improving Video Generation with Human Feedbackcited by 267
  6. 2024Diffusion Policy Policy Optimizationcited by 255
  7. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  8. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  9. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  10. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  11. 2024PuLID: Pure and Lightning ID Customization via Contrastive Alignmentcited by 143
  12. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
38 more
  1. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  2. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  3. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  4. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  5. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  6. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
  7. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  8. 2023Finetuning Text-to-Image Diffusion Models for Fairnesscited by 90
  9. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  10. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  11. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  12. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  13. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  14. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  15. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  16. 2024EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagationcited by 76
  17. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  18. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  19. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  20. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  21. 2024Gradient Guidance for Diffusion Models: An Optimization Perspectivecited by 59
  22. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Modelscited by 59
  23. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  24. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  25. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 59
  26. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  27. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  28. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
  29. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  30. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignmentcited by 51
  31. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  32. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  33. 2024DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the Generalised h-transformcited by 44
  34. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 41
  35. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  36. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  37. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  38. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38

50 most cited of 431 All 431 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
  2. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
  3. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  4. 2026MARBLE: Multi-Aspect Reward Balance for Diffusion RL
  5. 2026Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
  6. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning
  7. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  8. 2025Self-NPO: Data-Free Diffusion Model Enhancement via Truncated Diffusion Fine-Tuning