mlsanityData to 7 Oct 2026

Reinforcing Diffusion Models by Direct Group Preference Optimization

arXiv 2510.08425 ↗First posted 9 Oct 2025cs.LG cs.CVcited by 13

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

While reinforcement learning methods such as Group Relative Preference Optimization (GRPO) have significantly enhanced Large Language Models, adapting them to diffusion models remains challenging. In particular, GRPO demands a stochastic policy, yet the most cost-effective diffusion samplers are based on deterministic ODEs. Recent work addresses this issue by using inefficient SDE-based samplers to induce stochasticity, but this reliance on model-agnostic Gaussian noise leads to slow convergence. To resolve this conflict, we propose Direct Group Preference Optimization (DGPO), a new online RL algorithm that dispenses with the policy-gradient framework entirely. DGPO learns directly from group-level preferences, which utilize relative information of samples within groups. This design eliminates the need for inefficient stochastic policies, unlocking the use of efficient deterministic ODE samplers and faster training. Extensive results show that DGPO trains around 20 times faster than existing state-of-the-art methods and achieves superior performance on both in-domain and out-of-domain reward metrics. Code is available at https://github.com/Luo-Yihong/DGPO.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 50 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

45 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 41

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
29 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  5. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  6. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  8. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  9. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  10. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  11. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  12. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  13. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  14. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  15. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  16. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  17. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  18. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  19. 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
  20. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  21. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  22. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  23. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 139
  24. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  25. 2025Learning Few-Step Diffusion Models by Trajectory Distribution Matchingcited by 49
  26. 2024Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generationcited by 29
  27. 2025Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generationcited by 19
  28. 2025Adding Additional Control to One-Step Diffusion with Joint Distribution Matchingcited by 8
  29. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  2. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 11
  3. 2026TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Rewardcited by 7
  4. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 6
  5. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  6. 2026FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scalingcited by 2
  7. 2026Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matchingcited by 1
  8. 2026Advances in GRPO for Generation Models: A Surveycited by 1
  9. 2026Discrete Tilt Matchingcited by 0
  10. 2026Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hackingcited by 0
  11. 2026LeanGRPO: Eliminating Redundant Recomputation in Diffusion RLcited by 0
  12. 2026SGA-Flow-GRPO: Spatial Gradient-Guided Credit Assignment for Flow-GRPOcited by 0
1 more
  1. 2026ExploreNet: Learning Where to Explore in Diffusion GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
  2. 2024Group Robust Preference Optimization in Reward-free RLHF
  3. 2026Stabilizing Reinforcement Learning for Diffusion Language Models
  4. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policies
  5. 2026JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models
  6. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models
  7. 2025Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
  8. 2026UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models