mlsanityData to 7 Oct 2026

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning

arXiv 2602.17025 ↗First posted 19 Feb 2026cs.LGcited by 12

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 61 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

49 of 61 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy6 Aug
  2. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback4 Aug
  3. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning2 Jul
  4. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  5. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  8. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimization24 Feb
  9. 2025Process Reinforcement through Implicit Rewards3 Feb
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  11. 2024Free Process Rewards without Process Labels2 Dec
  12. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  10. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  11. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  12. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
25 more
  1. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  2. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  3. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  4. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 154
  5. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  6. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  7. 2019Weakly Supervised Disentanglement with Guaranteescited by 82
  8. 2016Controlling Output Length in Neural Encoder-Decoderscited by 75
  9. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 74
  10. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 65
  11. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  12. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  13. 2025Mitigating Overthinking in Large Reasoning Models via Manifold Steeringcited by 32
  14. 2025Activation Steering for Chain-of-Thought Compressioncited by 22
  15. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
  16. 2025Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamicscited by 16
  17. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 13
  18. 2024OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Modelscited by 12
  19. 2026SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenescited by 11
  20. 2025Listwise Preference Diffusion Optimization for User Behavior Trajectories Predictioncited by 10
  21. 2025In-context Ranking Preference Optimizationcited by 9
  22. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learningcited by 9
  23. 2025An Empirical Study on Zero-Data Bootstrapping for Conversational Recommender Systemscited by 5
  24. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  25. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 12
  2. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  3. 2026ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compressioncited by 2
  4. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  5. 2026Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Explorationcited by 1
  6. 2026Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneckcited by 1
  7. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
  8. 2026F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Rankingcited by 1
  9. 2026A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agentscited by 0
  10. 2026BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responsescited by 0
  11. 2026RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rolloutscited by 0
  12. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  2. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  3. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  4. 2026GRPO Training Dynamics for Small Language Models
  5. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  6. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  7. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  8. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models