mlsanityData to 14 Oct 2026

Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models

arXiv 2605.21123 ↗First posted 20 May 2026cs.CV cs.LGcited by 4

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Direct Preference Optimization (DPO) is successful for alignment in LLMs but still faces challenges in text-to-image generation. Existing studies are confined to denoising diffusion models while overlooking flow-matching, and suffer from an objective mismatch when applying discrete NLP-based DPO to regression-based generative tasks. In this paper, we derive a generalized DPO objective that covers both diffusion and flow-matching via a unified reverse-time SDE framework, and point out from a gradient perspective that the standard DPO objective is suboptimal for text-to-image generation. Consequently, we propose Linear-DPO, which replaces the aggressive sigmoid-based utility function with a sustained linear utility and incorporates an EMA-updated reference model. Qualitative and quantitative experiments on diffusion models (SD1.5, SDXL) and flow-matching model (SD3-Medium) demonstrate the superiority of our approach over existing baselines.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 36 of its 44 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

36 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models5 Nov
  2. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignment10 Jul
  3. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  5. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Reference10 Jun
  6. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
  7. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  8. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  9. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022Flow Matching for Generative Modelingcited by 5,702
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
15 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  7. 2025Qwen-Image Technical Reportcited by 1,044
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  11. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  12. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  13. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
  14. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  15. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 7
  2. 2026FAR-DPO: Feasibility-Aware and Robust Direct Preference Optimization for Cyclic Peptide Designcited by 0
  3. 2026Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hackingcited by 0
  4. 2026MEND: RL For Flow Models via Proximal Velocity Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment
  2. 2025Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models
  3. 2025Preference-Based Alignment of Discrete Diffusion Models
  4. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Models
  5. 2026Curriculum-DPO++: Direct Preference Optimization via Data and Model Curricula for Text-to-Image Generation
  6. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
  7. 2025Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
  8. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimization