mlsanityData to 14 Oct 2026

Large-scale Reinforcement Learning for Diffusion Models

arXiv 2401.12244 ↗First posted 20 Jan 2024cs.CV cs.AI cs.LGcited by 78

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Text-to-image diffusion models are a class of deep generative models that have demonstrated an impressive capacity for high-quality image generation. However, these models are susceptible to implicit biases that arise from web-scale text-image training pairs and may inaccurately model aspects of images we care about. This can result in suboptimal samples, model bias, and images that do not align with human ethics and preferences. In this paper, we present an effective scalable algorithm to improve diffusion models using Reinforcement Learning (RL) across a diverse set of reward functions, such as human preference, compositionality, and fairness over millions of images. We illustrate how our approach substantially outperforms existing methods for aligning diffusion models with human preferences. We further illustrate how this substantially improves pretrained Stable Diffusion (SD) models, generating samples that are preferred by humans 80.3% of the time over those from the base SD model while simultaneously improving both the composition and diversity of generated samples.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 39 of its 46 references that are in the map. First posted on arXiv 20 Jan 2024.

Builds on

39 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  2. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 37

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  12. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
25 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  2. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  3. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  5. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  6. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  7. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  8. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  9. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  10. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  11. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  12. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  13. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 226
  14. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 220
  15. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 151
  16. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  17. 2023Social Biases through the Text-to-Image Generation Lenscited by 127
  18. 2019Fair Generative Modeling via Weak Supervisioncited by 114
  19. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 97
  20. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 95
  21. 2023ITI-GEN: Inclusive Text-to-Image Generationcited by 64
  22. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesiscited by 51
  23. 2021Measuring Fairness in Generative Modelscited by 31
  24. 2023Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Setscited by 27
  25. 2023Mitigating stereotypical biases in text to image generative systemscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 39; 2026: 17.

78 papers.

Sort:

By area:

  1. 2025Improving Video Generation with Human Feedbackcited by 284
  2. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  3. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  4. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 87
  5. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 84
  6. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  7. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  8. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignmentcited by 53
  9. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 49
  10. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  11. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 42
  12. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 42
38 more
  1. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  2. 2024Reward Guided Latent Consistency Distillationcited by 35
  3. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  4. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  5. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  6. 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editorcited by 25
  7. 2025Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Modelscited by 23
  8. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  9. 2025Guiding Generative Models to Uncover Diverse and Novel Crystals via Reinforcement Learningcited by 21
  10. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignmentcited by 19
  11. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  12. 2024Towards Reliable Advertising Image Generation Using Human Feedbackcited by 16
  13. 2025Flow Map Distillation Without Datacited by 14
  14. 2024AdaNAT: Exploring Adaptive Policy for Token-Based Image Generationcited by 13
  15. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  16. 2024E-Motion: Future Motion Simulation via Event Sequence Diffusioncited by 12
  17. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 12
  18. 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generationcited by 12
  19. 2024Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learningcited by 11
  20. 2024ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generationcited by 11
  21. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 11
  22. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 11
  23. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 9
  24. 2025Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generationcited by 9
  25. 2025Steerable Scene Generation with Post Training and Inference-Time Searchcited by 9
  26. 2026Understanding Reward Hacking in Text-to-Image Reinforcement Learningcited by 9
  27. 2024Decomposed Direct Preference Optimization for Structure-Based Drug Designcited by 7
  28. 2025DiffDoctor: Diagnosing Image Diffusion Models Before Treatingcited by 7
  29. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  30. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 6
  31. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 6
  32. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  33. 2024Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Modelscited by 5
  34. 2025RL4Med-DDPO: Reinforcement Learning for Controlled Guidance Towards Diverse Medical Image Generation using Vision-Language Foundation Modelscited by 5
  35. 2025ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuningcited by 5
  36. 2026Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generationcited by 5
  37. 2024Latent Diffusion Models for Controllable RNA Sequence Generationcited by 4
  38. 2025AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Pathcited by 4

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
  2. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  3. 2023Training Diffusion Models with Reinforcement Learning
  4. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
  5. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation
  6. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learning
  7. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models
  8. 2023Diffusion Model Alignment Using Direct Preference Optimization