mlsanityData to 14 Oct 2026

Towards General Preference Alignment: Diffusion Models at Nash Equilibrium

arXiv 2605.04494 ↗First posted 6 May 2026cs.LG cs.CVcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has been popular for aligning text-to-image (T2I) diffusion models with human preferences. As a mainstream branch of RLHF, Direct Preference Optimization (DPO) offers a computationally efficient alternative that avoids explicit reward modeling and has been widely adopted in diffusion alignment. However, existing preference-based methods for diffusion alignment still rely on reward-induced preference signals and typically assume that human preferences can be adequately modeled by the Bradley--Terry (BT) model, which may fail to capture the full complexity of human preferences. In this paper, we formulate diffusion alignment from a game-theoretic perspective. We propose Diffusion Nash Preference Optimization (Diff.-NPO), an intuitive general preference framework for diffusion alignment. Diff.-NPO encourages the current policy to play against itself to achieve self improvement and lead to a better alignment. Empirically, we demonstrate the effectiveness of Diff.-NPO on the text-to-image generation task via various metrics. Diff.-NPO consistently outperforms existing preference-based diffusion alignment methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 37 of its 47 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

37 of 47 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Models6 Feb
  2. 2026SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedback5 Feb
  3. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Models2 Oct
  4. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Process19 Sep
  5. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance28 Aug
  6. 2025Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models16 May
  7. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  8. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  9. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Models4 Feb
  10. 2024Training-free Diffusion Model Alignment with Sampling Demons8 Oct
  11. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization10 Jun
  12. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
7 more
  1. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  2. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  3. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  4. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  6. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  7. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  9. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  10. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  11. 2023Nash Learning from Human Feedbackcited by 224
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
6 more
  1. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 93
  2. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  3. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  4. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  5. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 16
  6. 2025Improving Generative Ad Text on Facebook using Reinforcement Learningcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models
  2. 2025Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
  3. 2025Rethinking Direct Preference Optimization in Diffusion Models
  4. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
  5. 2026Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
  6. 2025BalancedDPO: Adaptive Multi-Metric Alignment
  7. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  8. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking