mlsanityData to 7 Oct 2026

Spurious Advantage Hidden in GRPO

arXiv 2609.04063 ↗First posted 3 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 52 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

29 of 52 references.

Closest ancestors · 21

Same direction, newest first

  1. 2026Your Group-Relative Advantage Is Biased13 Jan
  2. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting9 Oct
  3. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences8 Oct
  4. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives6 Oct
  5. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization6 Oct
  6. 2025It Takes Two: Your GRPO Is Secretly DPO1 Oct
  7. 2025Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models28 Sep
  8. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning26 Sep
  9. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems26 Sep
  10. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization23 Sep
  11. 2025MAPO: Mixed Advantage Policy Optimization23 Sep
  12. 2025Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR2 Sep
9 more
  1. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback4 Aug
  2. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
  3. 2025Geometric-Mean Policy Optimization28 Jul
  4. 2025BNPO: Beta Normalization Policy Optimization3 Jun
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 8

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  6. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  7. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learningcited by 32
  8. 2024MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkitcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  2. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  3. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation
  4. 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works
  5. 2026GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization with Enhanced Training Data Utilization for Sample-Constrained Reinforcement Learning
  6. 2026ReCo: Reweighting GRPO Against Distributional Concentration
  7. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  8. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization