mlsanityData to 7 Oct 2026

Aligning Diffusion Models by Optimizing Human Utility

arXiv 2404.04465 ↗First posted 6 Apr 2024cs.CVcited by 99

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 64 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

53 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  2. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  3. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  4. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  5. 2023Training Diffusion Models with Reinforcement Learning22 May
  6. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
35 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  13. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  14. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  15. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  16. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  17. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  18. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  19. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  20. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  21. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  22. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  23. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  24. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  25. 2022Teaching language models to support answers with verified quotescited by 290
  26. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  27. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  28. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  29. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  30. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  31. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  32. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  33. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  34. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  35. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generationcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 48; 2026: 39.

99 papers.

Sort:

By area:

  1. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  3. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  4. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  5. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  6. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  7. 2025A Survey of Direct Preference Optimizationcited by 35
  8. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  9. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  10. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  11. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  12. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
38 more
  1. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  2. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  3. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
  4. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27
  5. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  6. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignmentcited by 18
  7. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  8. 2024The Silent Assistant: NoiseQuery as Implicit Guidance for Goal-Driven Image Generationcited by 14
  9. 2025DreamFuse: Adaptive Image Fusion with Diffusion Transformercited by 13
  10. 2025DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolutioncited by 13
  11. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  12. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  13. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 12
  14. 2025Preference-Based Alignment of Discrete Diffusion Modelscited by 12
  15. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  16. 2024Aligning Diffusion Models with Noise-Conditioned Perceptioncited by 9
  17. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 9
  18. 2025RDPO: Real Data Preference Optimization for Physics Consistency Video Generationcited by 8
  19. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignmentcited by 8
  20. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  21. 2025Composition and Alignment of Diffusion Models using Constrained Learningcited by 8
  22. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimizationcited by 7
  23. 2025Steering Guidance for Personalized Text-to-Image Diffusion Modelscited by 7
  24. 2025Towards Better Optimization For Listwise Preference in Diffusion Modelscited by 7
  25. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Modelscited by 6
  26. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  27. 2025Discriminator-Free Direct Preference Optimization for Video Diffusioncited by 5
  28. 2025ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuningcited by 5
  29. 2026Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generationcited by 5
  30. 2026Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?cited by 5
  31. 2025Direct Distributional Optimization for Provable Alignment of Diffusion Modelscited by 4
  32. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
  33. 2025CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedbackcited by 4
  34. 2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
  35. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 4
  36. 2026Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimizationcited by 4
  37. 2026Lens: Rethinking Training Efficiency for Foundational Text-to-Image Modelscited by 4
  38. 2025SUDO: Enhancing Text-to-Image Diffusion Models with Self-Supervised Direct Preference Optimizationcited by 3

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preference
  2. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking
  3. 2025BalancedDPO: Adaptive Multi-Metric Alignment
  4. 2024Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models
  5. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  6. 2024Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback
  7. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization
  8. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference