mlsanityData to 14 Oct 2026

Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing

arXiv 2606.02218 ↗First posted 1 Jun 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Synchronous reinforcement learning methods such as Group Relative Policy Optimization (GRPO) provide stable and reproducible on-policy training, but they are highly vulnerable to stragglers, a single unusually long rollout can delay reward computation and parameter updates for the entire group. This problem becomes more severe as group size increases, creating a tension between the benefits of larger groups and the wall-clock cost of synchronization stalls. We propose Straggler-Aware Group Control (SAGC), a dynamic group-size controller that adapts the training group online based on observed rollout behavior. SAGC formulates group-size selection as an online constrained optimization problem, seeking to retain the benefits of larger groups while controlling the long-term rate of straggler events. Across synchronous GRPO and DAPO training, and on top of both vanilla and strong engineered baselines, SAGC consistently reduces straggler incidence and improves wall-clock efficiency while achieving competitive or better training reward. We further show that these gains transfer to final model quality: SAGC is competitive with or better than the strongest static group-size baseline on downstream reasoning benchmarks, and often produces shorter outputs without any explicit length penalty. These results position dynamic group control as a practical way to make synchronous on-policy RL more efficient and robust.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 30 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

23 of 30 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners30 Sep
  2. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training25 Sep
  3. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling4 Aug
  4. 2025Prefix Grouper: Efficient GRPO Training through Shared-Prefix Forward5 Jun
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Training Language Models to Reason Efficiently6 Feb
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2025Competitive Programming with Large Reasoning Modelscited by 119
  7. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 117
  8. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  9. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  10. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 82
  11. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  12. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
4 more
  1. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  2. 2025Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughtscited by 12
  3. 2026Efficient RLVR Training via Weighted Mutual Information Data Selectioncited by 6
  4. 2026SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillationcited by 1
  2. 2026Le Critique: Privileged Value Functions for LLM Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training
  2. 2025GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning
  3. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
  4. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
  5. 2026AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback
  6. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoning
  7. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
  8. 2026Constrained Group Relative Policy Optimization