mlsanityData to 14 Oct 2026

LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL

arXiv 2609.03528 ↗First posted 3 Sep 2026cs.LG cs.AI cs.ARcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Diffusion reinforcement learning (RL) has recently achieved significant success in post-training image and video generative models. However, most diffusion RL methods, including DanceGRPO and FlowGRPO, recompute selected timesteps with gradient tracking after rollout. Under on-policy training with the same backend for rollout and update, this recomputation is mathematically redundant. Intuitively, the rollout and policy update steps can reuse the same feed-forward backbone to avoid redundant computation, but doing so can incur a large memory overhead during rollout. To address the issue, we present LeanGRPO by restructuring the data-parallel layout and introducing two recompute-free training schedules for trajectory-logprob diffusion RL: (1) LeanGRPO-Retain enables gradient tracking during rollout and directly reuses the resulting computation graphs and saved activations for backward during update, requiring no recomputation; and (2) LeanGRPO-Reweight also enables gradients during rollout, but immediately backpropagates each selected step using a provisional advantage and delays gradient synchronization, then corrects the provisional gradients with the true advantage after the trajectory is completed. These schedules target different model scales and input sizes. Across FlowGRPO/DanceGRPO with FLUX.1-dev and Wan, LeanGRPO achieves up to 1.83x end-to-end speedup while preserving the original optimization objective.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 56 of its 62 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

56 of 62 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models9 Jun
  2. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models14 May
  3. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generation23 Mar
  4. 2026Diffusion Reinforcement Learning via Centered Reward Distillation14 Mar
  5. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models13 Mar
  6. 2026TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward8 Mar
  7. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design4 Feb
  8. 2025PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modeling2 Dec
  9. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation25 Nov
  10. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching7 Sep
  11. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning2 Mar
  12. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning3 Feb

Other references · 44

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  9. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  10. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  11. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  12. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
32 more
  1. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 97
  2. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 92
  3. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  4. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  5. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  6. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 32
  7. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Modelscited by 31
  8. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preferencecited by 30
  9. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Modelscited by 21
  10. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 21
  11. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 19
  12. 2025Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Samplingcited by 18
  13. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  14. 2025Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Modelscited by 16
  15. 2026vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Modelscited by 15
  16. 2025Reinforcing Diffusion Models by Direct Group Preference Optimizationcited by 14
  17. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 14
  18. 2026E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Modelscited by 14
  19. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Modelscited by 12
  20. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 10
  21. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 10
  22. 2026Stepwise Credit Assignment for GRPO on Flow-Matching Modelscited by 10
  23. 2026Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPOcited by 9
  24. 2026OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Modelscited by 7
  25. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  26. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 7
  27. 2026Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Trainingcited by 6
  28. 2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 5
  29. 2025PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Modelscited by 4
  30. 2026Faster and Better Alignment for Flow Matching Models via Step-aware Advantagescited by 4
  31. 2026Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimizationcited by 4
  32. 2026Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Compositioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
  2. 2025Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
  3. 2026DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
  4. 2026MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation
  5. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning
  6. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning
  7. 2026UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
  8. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models