mlsanityData to 14 Oct 2026

Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models

arXiv 2405.00760 ↗First posted 1 May 2024cs.CV cs.AIcited by 59

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Optimizing a text-to-image diffusion model with a given reward function is an important but underexplored research area. In this study, we propose Deep Reward Tuning (DRTune), an algorithm that directly supervises the final output image of a text-to-image diffusion model and back-propagates through the iterative sampling process to the input noise. We find that training earlier steps in the sampling process is crucial for low-level rewards, and deep supervision can be achieved efficiently and effectively by stopping the gradient of the denoising network input. DRTune is extensively evaluated on various reward models. It consistently outperforms other algorithms, particularly for low-level control signals, where all shallow supervision methods fail. Additionally, we fine-tune Stable Diffusion XL 1.0 (SDXL 1.0) model via DRTune to optimize Human Preference Score v2.1, resulting in the Favorable Diffusion XL 1.0 (FDXL 1.0) model. FDXL 1.0 significantly enhances image quality compared to SDXL 1.0 and reaches comparable quality compared with Midjourney v5.2.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 37 of its 43 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

37 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  3. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  4. 2023Training Diffusion Models with Reinforcement Learning22 May
  5. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2022Generative Adversarial Networkscited by 5,627
20 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2016Improved Techniques for Training GANscited by 2,970
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  11. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  12. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  13. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  14. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  15. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  16. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  17. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Qualitycited by 182
  18. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108
  19. 2022Controllable 3D Face Synthesis with Conditional Generative Occupancy Fieldscited by 37
  20. 2021Inverting Generative Adversarial Renderer for Face Reconstructioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 28; 2026: 22.

59 papers.

Sort:

By area:

  1. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
  2. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  3. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  4. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 50
  5. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 37
  6. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  7. 2024Improving GFlowNets for Text-to-Image Diffusion Alignmentcited by 27
  8. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27
  9. 2025Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Modelscited by 23
  10. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 19
  11. 2024Improving Long-Text Alignment for Text-to-Image Diffusion Modelscited by 18
  12. 2024Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewardcited by 16
38 more
  1. 2025Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Modelscited by 16
  2. 2025Video Generation Models Are Good Latent Reward Modelscited by 16
  3. 2024SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimizationcited by 15
  4. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 12
  5. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 11
  6. 2025Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generatorcited by 11
  7. 2025Composition and Alignment of Diffusion Models using Constrained Learningcited by 10
  8. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 9
  9. 2025ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Modelscited by 8
  10. 2025Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Modelscited by 8
  11. 2025Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Datacited by 7
  12. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimizationcited by 7
  13. 2025LongAnimation: Long Animation Generation with Dynamic Global-Local Memorycited by 6
  14. 2026Lookahead Sample Reward Guidance for Test-Time Scaling of Diffusion Modelscited by 6
  15. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  16. 2025High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioningcited by 5
  17. 2025ContentV: Efficient Training of Video Generation Models with Limited Computecited by 5
  18. 2025ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuningcited by 5
  19. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 5
  20. 2026Where Detectors Fail: Probing Generative Space for Generalizable AI-Generated Image Detectioncited by 5
  21. 2025ADT: Tuning Diffusion Models with Adversarial Supervisioncited by 4
  22. 2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
  23. 2026EasyTune: Efficient Step-Aware Fine-Tuning for Diffusion-Based Motion Generationcited by 4
  24. 2026AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimizationcited by 4
  25. 2026Efficient Adjoint Matching for Fine-tuning Diffusion Modelscited by 4
  26. 2025AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Modelscited by 3
  27. 2025PIRF: Physics-Informed Reward Fine-Tuning for Diffusion Modelscited by 2
  28. 2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
  29. 2025CogniEdit: Dense Gradient Flow Optimization for Fine-Grained Image Editingcited by 2
  30. 2026Reward Sharpness-Aware Fine-Tuning for Diffusion Modelscited by 2
  31. 2026Stitched Value Model for Diffusion Alignmentcited by 2
  32. 2024Low-Biased General Annotated Dataset Generationcited by 1
  33. 2025Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Imagecited by 1
  34. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuningcited by 1
  35. 2025RealMat: Realistic Materials with Diffusion and Reinforcement Learningcited by 1
  36. 2026MotionRFT: Unified Reinforcement Fine-Tuning for Text-to-Motion Generationcited by 1
  37. 2026Exploring the Design Space of Reward Backpropagation for Flow Matchingcited by 1
  38. 2026Latent Reward Registers for Diffusion Preference Alignmentcited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models
  2. 2024Reward Incremental Learning in Text-to-Image Generation
  3. 2026DiT-Reward: Generative Representations for Text-to-Image Reward Modeling
  4. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
  5. 2024Large-scale Reinforcement Learning for Diffusion Models
  6. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models
  7. 2026Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
  8. 2025A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models