mlsanityData to 7 Oct 2026

Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization

arXiv 2406.06382 ↗First posted 10 Jun 2024cs.CV cs.CL cs.LGcited by 30

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Aligning large language models with human preferences has emerged as a critical focus in language modeling research. Yet, integrating preference learning into Text-to-Image (T2I) generative models is still relatively uncharted territory. The Diffusion-DPO technique made initial strides by employing pairwise preference learning in diffusion models tailored for specific text prompts. We introduce Diffusion-RPO, a new method designed to align diffusion-based T2I models with human preferences more effectively. This approach leverages both prompt-image pairs with identical prompts and those with semantically related content across various modalities. Furthermore, we have developed a new evaluation metric, style alignment, aimed at overcoming the challenges of high costs, low reproducibility, and limited interpretability prevalent in current evaluations of human preference alignment. Our findings demonstrate that Diffusion-RPO outperforms established methods such as Supervised Fine-Tuning and Diffusion-DPO in tuning Stable Diffusion versions 1.5 and XL-1.0, achieving superior results in both automated evaluations of human preferences and style alignment. Our code is available at https://github.com/yigu1008/Diffusion-RPO

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 31 of its 46 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

31 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  2. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  3. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  4. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
15 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  5. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  8. 2015A Neural Algorithm of Artistic Stylecited by 480
  9. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  10. 2022Diffusion-GAN: Training GANs with Diffusioncited by 203
  11. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  12. 2021A Prototype-Oriented Framework for Unsupervised Domain Adaptationcited by 62
  13. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36
  14. 2020Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributionscited by 22
  15. 2023Beta Diffusioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 7.

30 papers.

Sort:

By area:

  1. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
  2. 2025A Survey of Direct Preference Optimizationcited by 35
  3. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  4. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  5. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  6. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27
  7. 2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 25
  8. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  9. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  10. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  11. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  12. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 9
18 more
  1. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 8
  2. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  3. 2024CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Modelscited by 7
  4. 2025Towards Better Optimization For Listwise Preference in Diffusion Modelscited by 7
  5. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
  6. 2025BalancedDPO: Adaptive Multi-Metric Alignmentcited by 6
  7. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  8. 2025Test-Time Preference Optimization for Image Restorationcited by 2
  9. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervisioncited by 2
  10. 2026Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulationcited by 2
  11. 2026PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learningcited by 2
  12. 2025When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPOcited by 1
  13. 2025Self-NPO: Data-Free Diffusion Model Enhancement via Truncated Diffusion Fine-Tuningcited by 1
  14. 2026ColorFLUX: A Structure-Color Decoupling Framework for Old Photo Colorizationcited by 1
  15. 2025PC-Diffusion: Aligning Diffusion Models with Human Preferences via Preference Classifiercited by 0
  16. 2026RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflectioncited by 0
  17. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibriumcited by 0
  18. 2026SwiftExplorer: Training-free Diffusion Model Alignment with Swift Diversity Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
  2. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignment
  3. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment
  4. 2024Aligning Diffusion Models with Noise-Conditioned Perception
  5. 2025Instant Preference Alignment for Text-to-Image Diffusion Models
  6. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Reference
  7. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Future
  8. 2025Rethinking Direct Preference Optimization in Diffusion Models