mlsanityData to 14 Oct 2026

Margin-aware Preference Optimization for Aligning Diffusion Models without Reference

arXiv 2406.06424 ↗First posted 10 Jun 2024cs.CVcited by 46

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Modern preference alignment methods, such as DPO, rely on divergence regularization to a reference model for training stability-but this creates a fundamental problem we call "reference mismatch." In this paper, we investigate the negative impacts of reference mismatch in aligning text-to-image (T2I) diffusion models, showing that larger reference mismatch hinders effective adaptation given the same amount of data, e.g., as when learning new artistic styles, or personalizing to specific objects. We demonstrate this phenomenon across text-to-image (T2I) diffusion models and introduce margin-aware preference optimization (MaPO), a reference-agnostic approach that breaks free from this constraint. By directly optimizing the likelihood margin between preferred and dispreferred outputs under the Bradley-Terry model without anchoring to a reference, MaPO transforms diverse T2I tasks into unified pairwise preference optimization. We validate MaPO's versatility across five challenging domains: (1) safe generation, (2) style adaptation, (3) cultural representation, (4) personalization, and (5) general preference alignment. Our results reveal that MaPO's advantage grows dramatically with reference mismatch severity, outperforming both DPO and specialized methods like DreamBooth while reducing training time by 15%. MaPO thus emerges as a versatile and memory-efficient method for generic T2I adaptation tasks.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 71 of its 104 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

71 of 104 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
  2. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  3. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  4. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  5. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  6. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  7. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  8. 2023Training Diffusion Models with Reinforcement Learning22 May
  9. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 62

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
38 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2021Variational Diffusion Modelscited by 1,153
  13. 2023LIMA: Less Is More for Alignmentcited by 1,100
  14. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  15. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  16. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  17. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  18. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  19. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  20. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  21. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  22. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  23. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  24. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  25. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  26. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  27. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
  28. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  29. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  30. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  31. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  32. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  33. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  34. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  35. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  36. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  37. 2024Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesiscited by 156
  38. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 23.

46 papers.

Sort:

By area:

  1. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  2. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 37
  3. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  4. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  5. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  6. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  7. 2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 26
  8. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Modelscited by 13
  9. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  10. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 12
  11. 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 10
  12. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 9
34 more
  1. 2024David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-trainingcited by 9
  2. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignmentcited by 8
  3. 2025Towards Better Optimization For Listwise Preference in Diffusion Modelscited by 7
  4. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Modelscited by 7
  5. 2025Length-Controlled Margin-Based Preference Optimization without Reference Modelcited by 6
  6. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  7. 2026Lens: Rethinking Training Efficiency for Foundational Text-to-Image Modelscited by 6
  8. 2026Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generationcited by 5
  9. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
  10. 2026CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Thinkcited by 4
  11. 2026Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimizationcited by 4
  12. 2026Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Modelscited by 4
  13. 2026SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedbackcited by 3
  14. 2026Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidancecited by 3
  15. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  16. 2025Test-Time Preference Optimization for Image Restorationcited by 2
  17. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervisioncited by 2
  18. 2026ColorFLUX: A Structure-Color Decoupling Framework for Old Photo Colorizationcited by 2
  19. 2026ViPO: Visual Preference Optimization at Scalecited by 2
  20. 2026Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimizationcited by 2
  21. 2025Aligning Diffusion Language Models via Unpaired Preference Optimizationcited by 1
  22. 2025Multi-dimensional Preference Alignment by Conditioning Reward Itselfcited by 1
  23. 2026Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairscited by 1
  24. 2026Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RLcited by 1
  25. 2026Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Modelscited by 1
  26. 2026FlowCPO: A Unified Divergence View of Preference Alignment for Flow Modelscited by 1
  27. 2025Dual Caption Preference Optimization for Diffusion Modelscited by 0
  28. 2025Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learningcited by 0
  29. 2026Personalizing Text-to-Image Generation to Individual Tastecited by 0
  30. 2026Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Modelscited by 0
  31. 2026Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hackingcited by 0
  32. 2026GuardPaint:SpeculativeSafetyDecodingforText-to-ImageGenerationcited by 0
  33. 2026ToPO: Token-Conditioned Preference Routing for Attention-Based Latent Diffusion Modelscited by 0
  34. 2026Geometry-Aware Preference Optimization for Text-to-Image Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking
  2. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization
  3. 2025BalancedDPO: Adaptive Multi-Metric Alignment
  4. 2024Aligning Diffusion Models with Noise-Conditioned Perception
  5. 2025Instant Preference Alignment for Text-to-Image Diffusion Models
  6. 2025PC-Diffusion: Aligning Diffusion Models with Human Preferences via Preference Classifier
  7. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
  8. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling