mlsanityData to 7 Oct 2026

Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning

arXiv 2508.20751 ↗First posted 28 Aug 2025cs.CVcited by 88

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advancements highlight the importance of GRPO-based reinforcement learning methods and benchmarking in enhancing text-to-image (T2I) generation. However, current methods using pointwise reward models (RM) for scoring generated images are susceptible to reward hacking. We reveal that this happens when minimal score differences between images are amplified after normalization, creating illusory advantages that drive the model to over-optimize for trivial gains, ultimately destabilizing the image generation process. To address this, we propose Pref-GRPO, a pairwise preference reward-based GRPO method that shifts the optimization objective from score maximization to preference fitting, ensuring more stable training. In Pref-GRPO, images are pairwise compared within each group using preference RM, and the win rate is used as the reward signal. Extensive experiments demonstrate that PREF-GRPO differentiates subtle image quality differences, providing more stable advantages and mitigating reward hacking. Additionally, existing T2I benchmarks are limited by coarse evaluation criteria, hindering comprehensive model assessment. To solve this, we introduce UniGenBench, a unified T2I benchmark comprising 600 prompts across 5 main themes and 20 subthemes. It evaluates semantic consistency through 10 primary and 27 sub-criteria, leveraging MLLM for benchmark construction and evaluation. Our benchmarks uncover the strengths and weaknesses of both open and closed-source T2I models and validate the effectiveness of Pref-GRPO.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 44 references that are in the map. First posted on arXiv 28 Aug 2025.

Builds on

38 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Models6 Aug
  2. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO22 May
  3. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  8. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  9. 2025Qwen-Image Technical Reportcited by 973
  10. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  11. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 939
21 more
  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  3. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  4. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  5. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  6. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  8. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  9. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  10. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  11. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  12. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  13. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  14. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  15. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  16. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  17. 2025Seedream 3.0 Technical Reportcited by 146
  18. 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
  19. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  20. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  21. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 65.

88 papers.

Sort:

By area:

  1. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  2. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
  3. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 33
  4. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preferencecited by 30
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  6. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 23
  7. 2025UniREditBench: A Unified Reasoning-based Image Editing Benchmarkcited by 23
  8. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 20
  9. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  10. 2025Flash-DMD: Towards High-Fidelity Few-Step Image Generation with Efficient Distillation and Joint Reinforcement Learningcited by 17
  11. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  12. 2026ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Rankingcited by 17
38 more
  1. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  2. 2025Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Modelscited by 16
  3. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 14
  4. 2026DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editingcited by 14
  5. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  6. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 13
  7. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Modelscited by 12
  8. 2026Unified Personalized Reward Model for Vision Generationcited by 11
  9. 2026GEditBench v2: A Human-Aligned Benchmark for General Image Editingcited by 11
  10. 2026ThinkRL-Edit: Thinking in Reinforcement Learning for Reasoning-Centric Image Editingcited by 10
  11. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Modelscited by 10
  12. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  13. 2026PosterOmni: Generalized Artistic Poster Creation via Task Distillation and Unified Reward Feedbackcited by 8
  14. 2026AI Can Learn Scientific Tastecited by 8
  15. 2026Leveraging Verifier-Based Reinforcement Learning in Image Editingcited by 8
  16. 2025PSR: Scaling Multi-Subject Personalized Image Generation with Pairwise Subject-Consistency Rewardscited by 7
  17. 2025RubricRL: Simple Generalizable Rewards for Text-to-Image Generationcited by 6
  18. 2026Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Modelscited by 6
  19. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  20. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 6
  21. 2025Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignmentcited by 5
  22. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 5
  23. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  24. 2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
  25. 2026Faster and Better Alignment for Flow Matching Models via Step-aware Advantagescited by 4
  26. 2026GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?cited by 4
  27. 2026AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimizationcited by 4
  28. 2026HP-Edit: A Human-Preference Post-Training Framework for Image Editingcited by 4
  29. 2026TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignmentcited by 4
  30. 2026Lens: Rethinking Training Efficiency for Foundational Text-to-Image Modelscited by 4
  31. 2025ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinementcited by 3
  32. 2026StruVis: Enhancing Reasoning-based Text-to-Image Generation via Thinking with Structured Visioncited by 3
  33. 2026From Sparse to Dense: Multi-View GRPO for Flow Models via Augmented Condition Spacecited by 3
  34. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  35. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 3
  36. 2026On the Adversarial Robustness of Multimodal LLM Judgescited by 3
  37. 2026Reward as An Agent for Embodied World Modelscited by 3
  38. 2026The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Workscited by 3

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
  2. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
  3. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
  4. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
  5. 2025Rethinking Reward Signals in Video GRPO: When Scores Become Targets
  6. 2026Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models
  7. 2026Understanding Reward Hacking in Text-to-Image Reinforcement Learning
  8. 2025From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation