mlsanityData to 7 Oct 2026

A Dense Reward View on Aligning Text-to-Image Diffusion with Preference

arXiv 2402.08265 ↗First posted 13 Feb 2024cs.CVcited by 55

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Aligning text-to-image diffusion model (T2I) with preference has been gaining increasing research attention. While prior works exist on directly optimizing T2I by preference data, these methods are developed under the bandit assumption of a latent reward on the entire diffusion reverse chain, while ignoring the sequential nature of the generation process. This may harm the efficacy and efficiency of preference alignment. In this paper, we take on a finer dense reward perspective and derive a tractable alignment objective that emphasizes the initial steps of the T2I reverse chain. In particular, we introduce temporal discounting into DPO-style explicit-reward-free objectives, to break the temporal symmetry therein and suit the T2I generation hierarchy. In experiments on single and multiple prompt generation, our method is competitive with strong relevant baselines, both quantitatively and qualitatively. Further investigations are conducted to illustrate the insight of our approach.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 92 of its 111 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

92 of 111 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  2. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  3. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  4. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  5. 2023Training Diffusion Models with Reinforcement Learning22 May
  6. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 86

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  13. 2020Learning to summarize from human feedbackcited by 2,562
  14. 2015Trust Region Policy Optimizationcited by 2,503
  15. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  16. 2017Hindsight Experience Replaycited by 1,096
  17. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  18. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  19. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  20. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  21. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  22. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  23. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  24. 2016Self-critical Sequence Training for Image Captioningcited by 544
  25. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  26. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  27. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  28. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  29. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  30. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  31. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  32. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  33. 2018Maximum a Posteriori Policy Optimisationcited by 318
  34. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  35. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  36. 2022Teaching language models to support answers with verified quotescited by 290
  37. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  38. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 21; 2026: 20.

55 papers.

Sort:

By area:

  1. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  2. 2024Reward Guided Latent Consistency Distillationcited by 35
  3. 2025A Survey of Direct Preference Optimizationcited by 35
  4. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  5. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  6. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  7. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
  8. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  9. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27
  10. 2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 25
  11. 2025RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learningcited by 22
  12. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 20
38 more
  1. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignmentcited by 18
  2. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 17
  3. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  4. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  5. 2025D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samplescited by 14
  6. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  7. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  8. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  9. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  10. 2024ByteEdit: Boost, Comply and Accelerate Generative Image Editingcited by 10
  11. 2025Few-Step Diffusion via Score identity Distillationcited by 10
  12. 2025Inversion-DPO: Precise and Efficient Post-Training for Diffusion Modelscited by 6
  13. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  14. 2024RFSR: Improving ISR Diffusion Models via Reward Feedback Learningcited by 5
  15. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Rankingcited by 5
  16. 2025TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimizationcited by 5
  17. 2025ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuningcited by 5
  18. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5
  19. 2025Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignmentcited by 5
  20. 2026Lens: Rethinking Training Efficiency for Foundational Text-to-Image Modelscited by 4
  21. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Surveycited by 3
  22. 2025Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedbackcited by 3
  23. 2026UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Modelscited by 3
  24. 2024Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimationcited by 2
  25. 2025Test-Time Preference Optimization for Image Restorationcited by 2
  26. 2026Alignment of Diffusion Model and Flow Matching for Text-to-Image Generationcited by 2
  27. 2026Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributionscited by 2
  28. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuningcited by 1
  29. 2026Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributescited by 1
  30. 2026Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidancecited by 1
  31. 2026Diffusion Controller: Framework, Algorithms and Parameterizationcited by 1
  32. 2026Threshold-Guided Optimization for Visual Generative Modelscited by 1
  33. 2026Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairscited by 1
  34. 2026Exploring the Design Space of Reward Backpropagation for Flow Matchingcited by 1
  35. 2026STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Trainingcited by 1
  36. 2026FlowCPO: A Unified Divergence View of Preference Alignment for Flow Modelscited by 1
  37. 2024DDIL: Diversity Enhancing Diffusion Distillation With Imitation Learningcited by 0
  38. 2025MEPG:Multi-Expert Planning and Generation for Compositionally-Rich Image Generationcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Diffusion Models by Optimizing Human Utility
  2. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
  3. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Models
  4. 2025BalancedDPO: Adaptive Multi-Metric Alignment
  5. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
  6. 2026Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
  7. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models
  8. 2024Improving Long-Text Alignment for Text-to-Image Diffusion Models