mlsanityData to 7 Oct 2026

Diffusion Model Alignment Using Direct Preference Optimization

arXiv 2311.12908 ↗First posted 21 Nov 2023cs.CV cs.AI cs.GRcited by 767

On the map: World models → World models for language agents

also in 2 more
Abstract (arXiv)

Large language models (LLMs) are fine-tuned using human comparison data with Reinforcement Learning from Human Feedback (RLHF) methods to make them better aligned with users' preferences. In contrast to LLMs, human preference learning has not been widely explored in text-to-image diffusion models; the best existing approach is to fine-tune a pretrained model using carefully curated high quality images and captions to improve visual appeal and text alignment. We propose Diffusion-DPO, a method to align diffusion models to human preferences by directly optimizing on human comparison data. Diffusion-DPO is adapted from the recently developed Direct Preference Optimization (DPO), a simpler alternative to RLHF which directly optimizes a policy that best satisfies human preferences under a classification objective. We re-formulate DPO to account for a diffusion model notion of likelihood, utilizing the evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with Diffusion-DPO. Our fine-tuned base model significantly outperforms both base SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement model in human evaluation, improving visual appeal and prompt alignment. We also develop a variant that uses AI feedback and has comparable performance to training on human preferences, opening the door for scaling of diffusion model alignment methods.

Key facts. Cited by 767 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 42 of its 58 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

42 of 58 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
30 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  13. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  14. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  15. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  16. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  17. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  18. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  19. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  20. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  21. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  22. 2020Rethinking Pre-training and Self-trainingcited by 352
  23. 2022Teaching language models to support answers with verified quotescited by 290
  24. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  25. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  26. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  27. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  28. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  29. 2022Defining and Characterizing Reward Hackingcited by 184
  30. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 127; 2025: 308; 2026: 331.

767 papers.

Sort:

By area:

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  2. 2025Qwen-Image Technical Reportcited by 973
  3. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  4. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 276
  5. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  6. 2025Improving Video Generation with Human Feedbackcited by 267
  7. 2024Diffusion Policy Policy Optimizationcited by 255
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  9. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  10. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  11. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  12. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
38 more
  1. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  2. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  3. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  4. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  5. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  6. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  7. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  8. 2025HunyuanVideo 1.5 Technical Reportcited by 148
  9. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  10. 2025HunyuanImage 3.0 Technical Reportcited by 138
  11. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  12. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  13. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  14. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  15. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  16. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  17. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  18. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  19. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  20. 2025LongCat-Image Technical Reportcited by 92
  21. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  22. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  23. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  24. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  25. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  26. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  27. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 81
  28. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  29. 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
  30. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  31. 20253D and 4D World Modeling: A Surveycited by 64
  32. 2025RewardDance: Reward Scaling in Visual Generationcited by 63
  33. 2024FakeInversion: Learning to Detect Images from Unseen Text-to-Image Models by Inverting Stable Diffusioncited by 61
  34. 2024Rectified Diffusion: Straightness Is Not Your Need in Rectified Flowcited by 59
  35. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  36. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  37. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  38. 2024Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attentioncited by 57

50 most cited of 767 All 767 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Direct Preference Optimization in Diffusion Models
  2. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  3. 2025Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
  4. 2024Large-scale Reinforcement Learning for Diffusion Models
  5. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference
  6. 2025Instant Preference Alignment for Text-to-Image Diffusion Models
  7. 2024Filtered Direct Preference Optimization
  8. 2024Aligning Crowd Feedback via Distributional Preference Reward Modeling