mlsanityData to 14 Oct 2026

MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE

arXiv 2507.21802 ↗First posted 29 Jul 2025cs.AI cs.CVcited by 178

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Although GRPO substantially enhances flow matching models in human preference alignment of image generation, methods such as FlowGRPO and DanceGRPO still exhibit inefficiency due to the necessity of sampling and optimizing over all denoising steps specified by the Markov Decision Process (MDP). In this paper, we propose MixGRPO, a novel framework that leverages the flexibility of mixed sampling strategies through the integration of stochastic differential equations (SDE) and ordinary differential equations (ODE). This streamlines the optimization process within the MDP to improve efficiency and boost performance. Specifically, MixGRPO introduces a sliding window mechanism, using SDE sampling and GRPO-guided optimization only within the window, while applying ODE sampling outside. This design confines sampling randomness to the time-steps within the window, thereby reducing the optimization overhead, and allowing for more focused gradient updates to accelerate convergence. Additionally, as time-steps beyond the sliding window are not involved in optimization, higher-order solvers are supported for faster sampling. So we present a faster variant, termed MixGRPO-Flash, which further improves training efficiency while achieving comparable performance. MixGRPO exhibits substantial gains across multiple dimensions of human preference alignment, outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50% lower training time. Notably, MixGRPO-Flash further reduces training time by 71%.

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 63 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

40 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  3. 2024Training-free Diffusion Model Alignment with Sampling Demons8 Oct
  4. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization6 Jun
  5. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimization29 May
  6. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  7. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  8. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  9. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2022Flow Matching for Generative Modelingcited by 5,702
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
19 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  4. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  5. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  6. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  7. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  8. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  9. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 450
  10. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  11. 2025Improving Video Generation with Human Feedbackcited by 284
  12. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  13. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  14. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  15. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  16. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 45
  17. 2019Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approachcited by 32
  18. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPOcited by 31
  19. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 44; 2026: 134.

178 papers.

Sort:

By area:

  1. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025HunyuanVideo 1.5 Technical Reportcited by 160
  4. 2025HunyuanImage 3.0 Technical Reportcited by 149
  5. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 92
  6. 2025LongCat-Video Technical Reportcited by 85
  7. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  8. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  9. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
  10. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  11. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 40
  12. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 34
38 more
  1. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Modelscited by 31
  2. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 29
  3. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Modelscited by 21
  4. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 21
  5. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 21
  6. 2025Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Samplingcited by 18
  7. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  8. 2025GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Modelscited by 17
  9. 2025Flash-DMD: Towards High-Fidelity Few-Step Image Generation with Efficient Distillation and Joint Reinforcement Learningcited by 17
  10. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  11. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  12. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Modelscited by 17
  13. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  14. 2025Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Modelscited by 16
  15. 2026TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Renderingcited by 15
  16. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 15
  17. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  18. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 14
  19. 2025Flow Map Distillation Without Datacited by 14
  20. 2026E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Modelscited by 14
  21. 2026Unified Personalized Reward Model for Vision Generationcited by 13
  22. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  23. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Modelscited by 12
  24. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generationcited by 12
  25. 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generationcited by 12
  26. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Modelscited by 12
  27. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 10
  28. 2025Seeing What Matters: Visual Preference Policy Optimization for Visual Generationcited by 10
  29. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  30. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 10
  31. 2025PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generationcited by 9
  32. 2026Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPOcited by 9
  33. 2026PosterOmni: Generalized Artistic Poster Creation via Task Distillation and Unified Reward Feedbackcited by 9
  34. 2025Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewardscited by 8
  35. 2026Reinforcing Few-step Generators via Reward-Tilted Distribution Matchingcited by 8
  36. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  37. 2025PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modelingcited by 7
  38. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
  2. 2026Stepwise Credit Assignment for GRPO on Flow-Matching Models
  3. 2025Value Gradient Guidance for Flow Matching Alignment
  4. 2026Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
  5. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
  6. 2026Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models
  7. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching
  8. 2026Drifting Preference Optimization for One-Step Generative Models