mlsanityData to 7 Oct 2026

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

arXiv 2311.13231 ↗First posted 22 Nov 2023cs.LG cs.AI cs.CVcited by 246

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Using reinforcement learning with human feedback (RLHF) has shown significant promise in fine-tuning diffusion models. Previous methods start by training a reward model that aligns with human preferences, then leverage RL techniques to fine-tune the underlying models. However, crafting an efficient reward model demands extensive datasets, optimal architecture, and manual hyperparameter tuning, making the process both time and cost-intensive. The direct preference optimization (DPO) method, effective in fine-tuning large language models, eliminates the necessity for a reward model. However, the extensive GPU memory requirement of the diffusion model's denoising process hinders the direct application of the DPO method. To address this issue, we introduce the Direct Preference for Denoising Diffusion Policy Optimization (D3PO) method to directly fine-tune diffusion models. The theoretical analysis demonstrates that although D3PO omits training a reward model, it effectively functions as the optimal reward model trained using human feedback data to guide the learning process. This approach requires no training of a reward model, proving to be more direct, cost-effective, and minimizing computational overhead. In experiments, our method uses the relative scale of objectives as a proxy for human preference, delivering comparable results to methods using ground-truth rewards. Moreover, D3PO demonstrates the ability to reduce image distortion rates and generate safer images, overcoming challenges lacking robust reward models. Our code is publicly available at https://github.com/yk7333/D3PO.

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 49 of its 66 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

49 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  2. 2023Training Diffusion Models with Reinforcement Learning22 May
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
33 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2015Variational Inference with Normalizing Flowscited by 2,161
  13. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  14. 2016Density estimation using Real NVPcited by 2,029
  15. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  16. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  17. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  18. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  19. 2016Pixel Recurrent Neural Networkscited by 814
  20. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  21. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  22. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  23. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  24. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  25. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  26. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  27. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  28. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  29. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  30. 2015Contextual Dueling Banditscited by 101
  31. 2022FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generationcited by 87
  32. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  33. 2015Data Generation as Sequential Decision Makingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 54; 2025: 108; 2026: 84.

246 papers.

Sort:

By area:

  1. 2025Improving Video Generation with Human Feedbackcited by 267
  2. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  3. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  4. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  5. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  6. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  7. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  8. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  9. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  10. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  11. 2024EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagationcited by 76
  12. 2024Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitivescited by 59
38 more
  1. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  2. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  3. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matchingcited by 55
  4. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  5. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
  6. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  7. 2024Schr"odinger Bridge Flow for Unpaired Data Translationcited by 47
  8. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  9. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  10. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 42
  11. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  12. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 38
  13. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  14. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  15. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35
  16. 2025A Survey of Direct Preference Optimizationcited by 35
  17. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  18. 2024Participation in the age of foundation modelscited by 32
  19. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  20. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  21. 2024Training-free Diffusion Model Alignment with Sampling Demonscited by 30
  22. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferencescited by 30
  23. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  24. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
  25. 2024Improving GFlowNets for Text-to-Image Diffusion Alignmentcited by 27
  26. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  27. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27
  28. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  29. 2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 25
  30. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
  31. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  32. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  33. 2025Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Modelscited by 21
  34. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 20
  35. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  36. 2025Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generationcited by 19
  37. 2024BATON: Aligning Text-to-Audio Model with Human Preference Feedbackcited by 18
  38. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignmentcited by 18

50 most cited of 246 All 246 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
  2. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning
  3. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards
  4. 2024New Desiderata for Direct Preference Optimization
  5. 2024Filtered Direct Preference Optimization
  6. 2024Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback
  7. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  8. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models