mlsanityData to 7 Oct 2026

DeDPO: Debiased Direct Preference Optimization for Diffusion Models

arXiv 2602.06195 ↗First posted 5 Feb 2026cs.CVcited by 1

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a predominant alignment method for diffusion models, facilitating off-policy training without explicit reward modeling. However, its reliance on large-scale, high-quality human preference labels presents a severe cost and scalability bottleneck. To overcome this, We propose a semi-supervised framework augmenting limited human data with a large corpus of unlabeled pairs annotated via cost-effective synthetic AI feedback. Our paper introduces Debiased DPO (DeDPO), which uniquely integrates a debiased estimation technique from causal inference into the DPO objective. By explicitly identifying and correcting the systematic bias and noise inherent in synthetic annotators, DeDPO ensures robust learning from imperfect feedback sources, including self-training and Vision-Language Models (VLMs). Experiments demonstrate that DeDPO is robust to the variations in synthetic labeling methods, achieving performance that matches and occasionally exceeds the theoretical upper bound of models trained on fully human-labeled data. This establishes DeDPO as a scalable solution for human-AI alignment using inexpensive synthetic supervision.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 57 of its 67 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

57 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models14 Oct
  2. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  3. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  4. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  5. 2023Training Diffusion Models with Reinforcement Learning22 May
  6. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  7. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  12. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  13. 2025Improving Video Generation with Human Feedbackcited by 267
  14. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  15. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  16. 2019Orthogonal Statistical Learningcited by 150
  17. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  18. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  19. 2022Semiparametric doubly robust targeted double machine learning: a reviewcited by 129
  20. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  21. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  22. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  23. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  24. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 52
  25. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  26. 2024Reward-Robust RLHF in LLMscited by 32
  27. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  28. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  29. 2025DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundnesscited by 24
  30. 2023TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wildcited by 22
  31. 2023Doubly Robust Self-Trainingcited by 18
  32. 2025Follow-Your-Preference: Towards Preference-Aligned Image Inpaintingcited by 18
  33. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  34. 2023Diffusion-SS3D: Diffusion Model for Semi-supervised 3D Object Detectioncited by 14
  35. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  36. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  37. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  38. 2025Doubly Robust Alignment for Large Language Modelscited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Panoptic Scene Program Diffusion Transformercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
  2. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  3. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  4. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
  5. 2024New Desiderata for Direct Preference Optimization
  6. 2026A Statistical Framework for Alignment with Biased AI Feedback
  7. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
  8. 2025Rethinking Direct Preference Optimization in Diffusion Models