mlsanityData to 7 Oct 2026

Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimation

arXiv 2411.14871 ↗First posted 22 Nov 2024cs.CV cs.AI cs.CLcited by 2

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion models have shown remarkable success in text-to-image generation, making preference alignment for these models increasingly important. The preference labels are typically available only at the terminal of denoising trajectories, which poses challenges in optimizing the intermediate denoising steps. In this paper, we propose to conduct Denoised Distribution Estimation (DDE) that explicitly connects intermediate steps to the terminal denoised distribution. Therefore, preference labels can be used for the entire trajectory optimization. To this end, we design two estimation strategies for our DDE. The first is stepwise estimation, which utilizes the conditional denoised distribution to estimate the model denoised distribution. The second is single-shot estimation, which converts the model output into the terminal denoised distribution via DDIM modeling. Analytically and empirically, we reveal that DDE equipped with two estimation strategies naturally derives a novel credit assignment scheme that prioritizes optimizing the middle part of the denoising trajectory. Extensive experiments demonstrate that our approach achieves superior performance, both quantitatively and qualitatively.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 48 of its 61 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

48 of 61 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
  2. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preference13 Feb
  3. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  4. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  6. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
30 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  10. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  13. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  14. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  15. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  16. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  17. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  18. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  19. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  20. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  21. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  22. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  23. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  24. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  25. 2024Direct Preference Optimization with an Offsetcited by 122
  26. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
  27. 2023Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Modelscited by 60
  28. 2022Understanding DDPM Latent Codes Through Optimal Transportcited by 44
  29. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  30. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontiercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
  2. 2026Explicit Critic Guidance for Aligning Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
  2. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision
  3. 2024Training-free Diffusion Model Alignment with Sampling Demons
  4. 2026Step-level Denoising-time Diffusion Alignment with Multiple Objectives
  5. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking
  6. 2026Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models
  7. 2025Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
  8. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignment