# Spurious Advantage Hidden in GRPO

arXiv: https://arxiv.org/abs/2609.04063 · HTML: https://mlsanity.com/paper/2609.04063/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)
- [MathScale: Scaling Instruction Tuning for Mathematical Reasoning](https://mlsanity.com/paper/2403.02884/) (2024)
- [Geometric-Mean Policy Optimization](https://mlsanity.com/paper/2507.20673/) (2025)
- [Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification](https://mlsanity.com/paper/2503.06639/) (2025)
- [EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity](https://mlsanity.com/paper/2507.21848/) (2025)
- [R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning](https://mlsanity.com/paper/2506.04185/) (2025)
- [Your Group-Relative Advantage Is Biased](https://mlsanity.com/paper/2601.08521/) (2026)
- [NGRPO: Negative-enhanced Group Relative Policy Optimization](https://mlsanity.com/paper/2509.18851/) (2025)
- [BNPO: Beta Normalization Policy Optimization](https://mlsanity.com/paper/2506.02864/) (2025)
- [It Takes Two: Your GRPO Is Secretly DPO](https://mlsanity.com/paper/2510.00977/) (2025)
- [Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives](https://mlsanity.com/paper/2510.04996/) (2025)
- [MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems](https://mlsanity.com/paper/2509.22047/) (2025)
- [DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization](https://mlsanity.com/paper/2510.04474/) (2025)
- [MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkit](https://mlsanity.com/paper/2404.13925/) (2024)
- [Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting](https://mlsanity.com/paper/2510.08696/) (2025)
- [TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2508.02833/) (2025)
- [MAPO: Mixed Advantage Policy Optimization](https://mlsanity.com/paper/2509.18849/) (2025)
- [Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning](https://mlsanity.com/paper/2509.22611/) (2025)
- [λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences](https://mlsanity.com/paper/2510.06870/) (2025)
- [Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR](https://mlsanity.com/paper/2509.02522/) (2025)
- [Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models](https://mlsanity.com/paper/2509.23962/) (2025)

## Abstract (arXiv)

Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.