mlsanityData to 7 Oct 2026

Curriculum Direct Preference Optimization for Diffusion and Consistency Models

arXiv 2405.13637 ↗First posted 22 May 2024cs.CV cs.AI cs.LGcited by 36

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Direct Preference Optimization (DPO) has been proposed as an effective and efficient alternative to reinforcement learning from human feedback (RLHF). In this paper, we propose a novel and enhanced version of DPO based on curriculum learning for text-to-image generation. Our method is divided into two training stages. First, a ranking of the examples generated for each prompt is obtained by employing a reward model. Then, increasingly difficult pairs of examples are sampled and provided to a text-to-image generative (diffusion or consistency) model. Generated samples that are far apart in the ranking are considered to form easy pairs, while those that are close in the ranking form hard pairs. In other words, we use the rank difference between samples as a measure of difficulty. The sampled pairs are split into batches according to their difficulty levels, which are gradually used to train the generative model. Our approach, Curriculum DPO, is compared against state-of-the-art fine-tuning approaches on nine benchmarks, outperforming the competing methods in terms of text alignment, aesthetics and human preference. Our code is available at https://github.com/CroitoruAlin/Curriculum-DPO.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 67 of its 83 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

67 of 83 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  2. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  3. 2023Training Diffusion Models with Reinforcement Learning22 May
  4. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  13. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  14. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  15. 2023Consistency Modelscited by 1,778
  16. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  17. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  18. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  19. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  20. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  21. 2021Variational Diffusion Modelscited by 1,145
  22. 2022Diffusion Models in Vision: A Surveycited by 1,008
  23. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  24. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  25. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  26. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  27. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  28. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  29. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  30. 2021Score-based Generative Modeling in Latent Spacecited by 634
  31. 2019Green AIcited by 620
  32. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  33. 2023Improved Techniques for Training Consistency Modelscited by 430
  34. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  35. 2021Score-based diffusion models for accelerated MRIcited by 308
  36. 2021Curriculum Learning: A Surveycited by 257
  37. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  38. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 19.

36 papers.

Sort:

By area:

  1. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  2. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  3. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 22
  4. 2024Boost Your Human Image Generation Model via Direct Preference Optimizationcited by 16
  5. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 14
  6. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  7. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  8. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  9. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 8
  10. 2025Dual-Difficulty Curriculum Learning for Direct Preference Optimizationcited by 5
  11. 2025Improving LLMs for Machine Translation Using Synthetic Preference Datacited by 2
  12. 2025Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generationcited by 2
24 more
  1. 2025Test-Time Preference Optimization for Image Restorationcited by 2
  2. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervisioncited by 2
  3. 2026Small-Margin Preferences Still Matter-If You Train Them Rightcited by 2
  4. 2026Drifting Preference Optimization for One-Step Generative Modelscited by 2
  5. 2025When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPOcited by 1
  6. 2025Sell It Before You Make It: Revolutionizing E-Commerce with Personalized AI-Generated Itemscited by 1
  7. 2025From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanationscited by 1
  8. 2026CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learningcited by 1
  9. 2026Human detectors are surprisingly powerful reward modelscited by 1
  10. 2026Diffusion Controller: Framework, Algorithms and Parameterizationcited by 1
  11. 2026POCA: Pareto-Optimal Curriculum Alignment for Visual Text Generationcited by 1
  12. 2026Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairscited by 1
  13. 2026SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learningcited by 1
  14. 2025g-DPO: Scalable Preference Optimization for Protein Language Modelscited by 0
  15. 2026SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalizationcited by 0
  16. 2026Region-Normalized DPO for Medical Image Segmentation under Noisy Judgescited by 0
  17. 2026Curriculum-DPO++: Direct Preference Optimization via Data and Model Curricula for Text-to-Image Generationcited by 0
  18. 2026Learning Hierarchical and Geometry-Aware Graph Representations for Text-to-CADcited by 0
  19. 2026Distributed Direct Preference Optimizationcited by 0
  20. 2026Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Developmentcited by 0
  21. 2026Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimizationcited by 0
  22. 2026ToPO: Token-Conditioned Preference Routing for Attention-Based Latent Diffusion Modelscited by 0
  23. 2026CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generationcited by 0
  24. 2026FestDPO: Few-step Generator Alignment with Direct Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
  2. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences
  3. 2024Scalable Ranked Preference Optimization for Text-to-Image Generation
  4. 2026Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generation
  5. 2026Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models
  6. 2024Filtered Direct Preference Optimization
  7. 2025Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
  8. 2024SEE-DPO: Self Entropy Enhanced Direct Preference Optimization