mlsanityData to 14 Oct 2026

DanceGRPO: Unleashing GRPO on Visual Generation

arXiv 2505.07818 ↗First posted 12 May 2025cs.CVcited by 413

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Recent advances in generative AI have revolutionized visual content creation, yet aligning model outputs with human preferences remains a critical challenge. While Reinforcement Learning (RL) has emerged as a promising approach for fine-tuning generative models, existing methods like DDPO and DPOK face fundamental limitations - particularly their inability to maintain stable optimization when scaling to large and diverse prompt sets, severely restricting their practical utility. This paper presents DanceGRPO, a framework that addresses these limitations through an innovative adaptation of Group Relative Policy Optimization (GRPO) for visual generation tasks. Our key insight is that GRPO's inherent stability mechanisms uniquely position it to overcome the optimization challenges that plague prior RL-based approaches on visual generation. DanceGRPO establishes several significant advances: First, it demonstrates consistent and stable policy optimization across multiple modern generative paradigms, including both diffusion models and rectified flows. Second, it maintains robust performance when scaling to complex, real-world scenarios encompassing three key tasks and four foundation models. Third, it shows remarkable versatility in optimizing for diverse human preferences as captured by five distinct reward models assessing image/video aesthetics, text-image alignment, video motion quality, and binary feedback. Our comprehensive experiments reveal that DanceGRPO outperforms baseline methods by up to 181% across multiple established benchmarks, including HPS-v2.1, CLIP Score, VideoAlign, and GenEval. Our results establish DanceGRPO as a robust and versatile solution for scaling Reinforcement Learning from Human Feedback (RLHF) tasks in visual generation, offering new insights into harmonizing reinforcement learning and visual synthesis.

Key facts. Cited by 413 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 57 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

30 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
16 more
  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  2. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  3. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  4. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 287
  5. 2025Improving Video Generation with Human Feedbackcited by 284
  6. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  7. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  8. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  9. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 131
  10. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 131
  11. 2024Video Diffusion Alignment via Reward Gradientscited by 107
  12. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 84
  13. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 62
  14. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 41
  15. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 37
  16. 2024UniFL: Improve Latent Diffusion Model via Unified Feedback Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 138; 2026: 275.

413 papers.

Sort:

By area:

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  2. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 259
  3. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  4. 2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 249
  5. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  6. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
  7. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  8. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  10. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
  11. 2026Seedance 2.0: Advancing Video Generation for World Complexitycited by 160
  12. 2025LongCat-Image Technical Reportcited by 100
38 more
  1. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 97
  2. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 92
  3. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
  4. 2025LongCat-Video Technical Reportcited by 85
  5. 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Modelcited by 80
  6. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  7. 2025RewardDance: Reward Scaling in Visual Generationcited by 68
  8. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  9. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  10. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 64
  11. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  12. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matchingcited by 56
  13. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
  14. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 50
  15. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  16. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 40
  17. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 40
  18. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  19. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  20. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  21. 2025BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanationcited by 33
  22. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPOcited by 31
  23. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 31
  24. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Modelscited by 31
  25. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preferencecited by 30
  26. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 29
  27. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 25
  28. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
  29. 2026WorldCompass: Reinforcement Learning for Long-Horizon World Modelscited by 24
  30. 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Rewardcited by 24
  31. 2025RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learningcited by 23
  32. 2025OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learningcited by 23
  33. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learningcited by 22
  34. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 22
  35. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 22
  36. 2025One-Step Flow Policy Mirror Descentcited by 21
  37. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Modelscited by 21
  38. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 21

50 most cited of 413 All 413 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization
  2. 2026Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
  3. 2025Flow-GRPO: Training Flow Matching Models via Online RL
  4. 2026From Broad Exploration to Stable Synthesis: Entropy-Guided Optimization for Autoregressive Image Generation
  5. 2026Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation
  6. 2025From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation
  7. 2025PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models
  8. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective