mlsanityData to 14 Oct 2026

BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models

arXiv 2509.06040 ↗First posted 7 Sep 2025cs.CV cs.AI cs.LGcited by 61

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Recent progress in aligning image and video generative models with Group Relative Policy Optimization (GRPO) has improved human preference alignment, but existing variants remain inefficient due to sequential rollouts and large numbers of sampling steps, unreliable credit assignment: sparse terminal rewards are uniformly propagated across timesteps, failing to capture the varying criticality of decisions during denoising. In this paper, we present BranchGRPO, a method that restructures the rollout process into a branching tree, where shared prefixes amortize computation and pruning removes low-value paths and redundant depths. BranchGRPO introduces three contributions: (1) a branching scheme that amortizes rollout cost through shared prefixes while preserving exploration diversity; (2) a reward fusion and depth-wise advantage estimator that transforms sparse terminal rewards into dense step-level signals; and (3) pruning strategies that cut gradient computation but leave forward rollouts and exploration unaffected. On HPDv2.1 image alignment, BranchGRPO improves alignment scores by up to 16% over DanceGRPO, while reducing per-iteration training time by nearly 55%. A hybrid variant, BranchGRPO-Mix, further accelerates training to 4.7x faster than DanceGRPO without degrading alignment. On WanX video generation, it further achieves higher Video-Align scores with sharper and temporally consistent frames compared to DanceGRPO. Codes are available at hrefhttps://fredreic1849.github.io/BranchGRPO-Webpage/BranchGRPO.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 34 of its 41 references that are in the map. First posted on arXiv 7 Sep 2025.

Builds on

34 of 41 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Models6 Aug
  2. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE29 Jul
  3. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  5. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization6 Jun
  6. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation15 Feb
  7. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  8. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  9. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  10. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  11. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  10. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
11 more
  1. 2018Demystifying MMD GANscited by 1,293
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  3. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  4. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  5. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  6. 2025Improving Video Generation with Human Feedbackcited by 284
  7. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  8. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  9. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
  10. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 61
  11. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 49.

61 papers.

Sort:

By area:

  1. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  2. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  3. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  4. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  5. 2025Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Modelscited by 16
  6. 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoderscited by 15
  7. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 15
  8. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 14
  9. 2026E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Modelscited by 14
  10. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Modelscited by 12
  11. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoningcited by 12
  12. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 10
38 more
  1. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  2. 2026Video Models Can Reason with Verifiable Rewardscited by 9
  3. 2025Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewardscited by 8
  4. 2025PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modelingcited by 7
  5. 2026OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Modelscited by 7
  6. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  7. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  8. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 7
  9. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 6
  10. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  11. 2025ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinementcited by 5
  12. 2026Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generationcited by 5
  13. 2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 5
  14. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 5
  15. 2026A Systematic Post-Train Framework for Video Generationcited by 5
  16. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  17. 2025SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Modelscited by 4
  18. 2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
  19. 2026AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimizationcited by 4
  20. 2026TempAct: Advancing Temporal Plausibility in Autoregressive Video Generation via Planner-Executor RLcited by 4
  21. 2026From Sparse to Dense: Multi-View GRPO for Flow Models via Augmented Condition Spacecited by 3
  22. 2026FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scalingcited by 3
  23. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  24. 2026When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropycited by 3
  25. 2026Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Modelscited by 3
  26. 2026Scaling Reinforcement Learning for Diffusion Models via Velocity Matchingcited by 3
  27. 2026Talk2Move: Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenescited by 2
  28. 2026Advances in GRPO for Generation Models: A Surveycited by 2
  29. 2026Guiding Distribution Matching Distillation with Gradient-Based Reinforcement Learningcited by 2
  30. 2026Stitched Value Model for Diffusion Alignmentcited by 2
  31. 2026Faster Synchronous On-Policy RL via Straggler-Aware Group Sizingcited by 2
  32. 2026AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPOcited by 2
  33. 2025C2DLM: Causal Concept-Guided Diffusion Large Language Modelscited by 1
  34. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Modelscited by 1
  35. 2026DRM: Diffusion-based Reward Model With Step-wise Guidancecited by 1
  36. 2026AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Modelscited by 1
  37. 2026OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learningcited by 1
  38. 2026Step Back to Move Forward: Reflection-Aware Preference Optimization for Visual Generationcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
  2. 2026Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization
  3. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment
  4. 2025Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
  5. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Models
  6. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models
  7. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
  8. 2025Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Models