mlsanityData to 7 Oct 2026

Scalable Ranked Preference Optimization for Text-to-Image Generation

arXiv 2410.18013 ↗First posted 23 Oct 2024cs.CVcited by 33

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a powerful approach to align text-to-image (T2I) models with human feedback. Unfortunately, successful application of DPO to T2I models requires a huge amount of resources to collect and label large-scale datasets, e.g., millions of generated paired images annotated with human preferences. In addition, these human preference datasets can get outdated quickly as the rapid improvements of T2I models lead to higher quality images. In this work, we investigate a scalable approach for collecting large-scale and fully synthetic datasets for DPO training. Specifically, the preferences for paired images are generated using a pre-trained reward function, eliminating the need for involving humans in the annotation process, greatly improving the dataset collection efficiency. Moreover, we demonstrate that such datasets allow averaging predictions across multiple models and collecting ranked preferences as opposed to pairwise preferences. Furthermore, we introduce RankDPO to enhance DPO-based methods using the ranking feedback. Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated preference dataset "Syn-Pic" improves both prompt-following (on benchmarks like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user studies). This pipeline presents a practical and scalable solution to develop better preference datasets to enhance the performance of text-to-image models.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 72 of its 92 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

72 of 92 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models9 Sep
  2. 2024Video Diffusion Alignment via Reward Gradients11 Jul
  3. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Reference10 Jun
  4. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization10 Jun
  5. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization6 Jun
  6. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation23 Apr
  7. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
  8. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching4 Apr
  9. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models13 Feb
  10. 2023Rich Human Feedback for Text-to-Image Generation15 Dec
  11. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  12. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
1 more
  1. 2023Aligning Text-to-Image Models using Human Feedback23 Feb

Other references · 59

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Flow Matching for Generative Modelingcited by 5,426
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  9. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  11. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  13. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  14. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  15. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  16. 2016Generative Adversarial Text to Image Synthesiscited by 735
  17. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  18. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  19. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  20. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  21. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  22. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  23. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  24. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  25. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  26. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  27. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  28. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  29. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  30. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  31. 2021Simple multi-dataset detectioncited by 126
  32. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  33. 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 102
  34. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
  35. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  36. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  37. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  38. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 66

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 15.

33 papers.

Sort:

By area:

  1. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  2. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35
  3. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  4. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  5. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 22
  6. 2025Aligning Text to Image in Diffusion Models is Easier Than You Thinkcited by 21
  7. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  8. 2024A Statistical Framework for Ranking LLM-Based Chatbotscited by 11
  9. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  10. 2025BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignmentcited by 9
  11. 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 8
  12. 2024CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Modelscited by 7
21 more
  1. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimizationcited by 7
  2. 2025Towards Better Optimization For Listwise Preference in Diffusion Modelscited by 7
  3. 2025DesignPref: Capturing Personal Preferences in Visual Design Generationcited by 6
  4. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  5. 2025Semi-Supervised Preference Optimization with Limited Feedbackcited by 3
  6. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  7. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairscited by 2
  8. 2025Stitch: Training-Free Position Control in Multimodal Diffusion Transformerscited by 2
  9. 2026Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimizationcited by 2
  10. 2025Sell It Before You Make It: Revolutionizing E-Commerce with Personalized AI-Generated Itemscited by 1
  11. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  12. 2026EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesiscited by 1
  13. 2026ViPO: Visual Preference Optimization at Scalecited by 1
  14. 2026Threshold-Guided Optimization for Visual Generative Modelscited by 1
  15. 2026Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairscited by 1
  16. 2026Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Modelscited by 1
  17. 2026Di3PO - Diptych Diffusion DPO for Targeted Improvements in Image Generationcited by 0
  18. 2026Personalizing Text-to-Image Generation to Individual Tastecited by 0
  19. 2026Efficient Personalization of Generative User Interfacescited by 0
  20. 2026Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Modelscited by 0
  21. 2026Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
  2. 2025HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment
  3. 2025BalancedDPO: Adaptive Multi-Metric Alignment
  4. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference
  5. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Models
  6. 2026Drifting Preference Optimization for One-Step Generative Models
  7. 2026Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
  8. 2025When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPO