mlsanityData to 7 Oct 2026

Aligning Text-to-Image Diffusion Models with Reward Backpropagation

arXiv 2310.03739 ↗First posted 5 Oct 2023cs.CV cs.AI cs.LGcited by 284

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Text-to-image diffusion models have recently emerged at the forefront of image generation, powered by very large-scale unsupervised or weakly supervised text-to-image training datasets. Due to their unsupervised training, controlling their behavior in downstream tasks, such as maximizing human-perceived image quality, image-text alignment, or ethical image generation, is difficult. Recent works finetune diffusion models to downstream reward functions using vanilla reinforcement learning, notorious for the high variance of the gradient estimators. In this paper, we propose AlignProp, a method that aligns diffusion models to downstream reward functions using end-to-end backpropagation of the reward gradient through the denoising process. While naive implementation of such backpropagation would require prohibitive memory resources for storing the partial derivatives of modern text-to-image models, AlignProp finetunes low-rank adapter weight modules and uses gradient checkpointing, to render its memory usage viable. We test AlignProp in finetuning diffusion models to various objectives, such as image-text semantic alignment, aesthetics, compressibility and controllability of the number of objects present, as well as their combinations. We show AlignProp achieves higher rewards in fewer training steps than alternatives, while being conceptually simpler, making it a straightforward choice for optimizing diffusion models for differentiable reward functions of interest. Code and Visualization results are available at https://align-prop.github.io/.

Key facts. Cited by 284 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 45 of its 55 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

45 of 55 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  2. 2023Training Diffusion Models with Reinforcement Learning22 May
  3. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  5. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
26 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  10. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  11. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  13. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  14. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  15. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  16. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  17. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  18. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  20. 2022Latent Diffusion for Language Generationcited by 110
  21. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107
  22. 2023Language Conditioned Traffic Generationcited by 86
  23. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  24. 2016Memory-Efficient Backpropagation Through Timecited by 78
  25. 2017Unbiasing Truncated Backpropagation Through Timecited by 37
  26. 2017Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Netcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 69; 2025: 107; 2026: 101.

284 papers.

Sort:

By area:

  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  3. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  4. 2025Improving Video Generation with Human Feedbackcited by 267
  5. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  6. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  7. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  8. 2024PuLID: Pure and Lightning ID Customization via Contrastive Alignmentcited by 143
  9. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  10. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  11. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  12. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
38 more
  1. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  2. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  3. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  4. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  5. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  6. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  7. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  8. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  9. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  10. 2024EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagationcited by 76
  11. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  12. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  13. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  14. 2024Gradient Guidance for Diffusion Models: An Optimization Perspectivecited by 59
  15. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Modelscited by 59
  16. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  17. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  18. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  19. 2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
  20. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  21. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  22. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  23. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 41
  24. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  25. 2024Reward Guided Latent Consistency Distillationcited by 35
  26. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35
  27. 2025Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenizationcited by 35
  28. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  29. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  30. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  31. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  32. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  33. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  34. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  35. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learningcited by 30
  36. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimizationcited by 30
  37. 2024Training-free Diffusion Model Alignment with Sampling Demonscited by 30
  38. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferencescited by 30

50 most cited of 284 All 284 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion Models
  2. 2024Large-scale Reinforcement Learning for Diffusion Models
  3. 2025A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models
  4. 2024Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback
  5. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  6. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models
  7. 2026Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusion
  8. 2025ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal Diffusion