mlsanityData to 7 Oct 2026

On the Hidden Objective Biases of Group-based Reinforcement Learning

arXiv 2601.05002 ↗First posted 8 Jan 2026cs.LG cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group-based reinforcement learning methods, like Group Relative Policy Optimization (GRPO), are widely used nowadays to post-train large language models. Despite their empirical success, they exhibit structural mismatches between reward optimization and the underlying training objective. In this paper, we present a theoretical analysis of GRPO style methods by studying them within a unified surrogate formulation. This perspective reveals recurring properties that affect all the methods under analysis: (i) non-uniform group weighting induces systematic gradient biases on shared prefix tokens; (ii) interactions with the AdamW optimizer make training dynamics largely insensitive to reward scaling; and (iii) optimizer momentum can push policy updates beyond the intended clipping region under repeated optimization steps. We believe that these findings highlight fundamental limitations of current approaches and provide principled guidance for the design of future formulations.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 18 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

12 of 18 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025GCPO: When Contrast Fails, Go Gold9 Oct
  2. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning5 Oct
  3. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems26 Sep
  4. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training28 May
  5. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  6. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 4

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  3. 2016Learning values across many orders of magnitudecited by 80
  4. 2025Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answerscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignmentcited by 0
  2. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  2. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  3. 2026On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
  4. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  5. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  6. 2026Stabilizing Reinforcement Learning for Diffusion Language Models
  7. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
  8. 2026Group Entropy-Controlled Policy Optimization