mlsanityData to 7 Oct 2026

Your Group-Relative Advantage Is Biased

arXiv 2601.08521 ↗First posted 13 Jan 2026cs.LGcited by 32

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 40 references that are in the map. First posted on arXiv 13 Jan 2026.

Builds on

25 of 40 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization6 Dec
  2. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning26 Oct
  3. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning1 Oct
  4. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning29 Sep
  5. 2025Geometric-Mean Policy Optimization28 Jul
  6. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning22 May
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2023Let's Verify Step by Step31 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 16

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  7. 2019On Mutual Information Maximization for Representation Learningcited by 315
  8. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  9. 2019Cross-Batch Memory for Embedding Learningcited by 130
  10. 2024LLM-Forest: Ensemble Learning of LLMs with Graph-Augmented Prompts for Data Imputationcited by 25
  11. 2021Cross-Batch Negative Sampling for Training Two-Tower Recommenderscited by 24
  12. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
4 more
  1. 2020Cross-Iteration Batch Normalizationcited by 20
  2. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  3. 2025IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinementcited by 3
  4. 2025A domain-theoretic framework for conditional probability and Bayesian updating in programmingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 32.

32 papers.

Sort:

By area:

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  2. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 17
  3. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 13
  4. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  5. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 11
  6. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 11
  7. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 10
  8. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  9. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  10. 2026Weak-Driven Learning: How Weak Agents make Strong Agents Strongercited by 5
  11. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rolloutscited by 5
  12. 2026When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Modelscited by 5
20 more
  1. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
  2. 2026GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignmentcited by 4
  3. 2026Adaptive Robust Estimator for Multi-Agent Reinforcement Learningcited by 2
  4. 2026Policy Improvement Reinforcement Learningcited by 2
  5. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1
  6. 2026Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoningcited by 1
  7. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  8. 2026On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewardscited by 1
  9. 2026Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rolloutscited by 1
  10. 2026CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoningcited by 0
  11. 2026Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregationcited by 0
  12. 2026LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learningcited by 0
  13. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  14. 2026WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Trainingcited by 0
  15. 2026Spurious Advantage Hidden in GRPOcited by 0
  16. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 0
  17. 2026TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMscited by 0
  18. 2026SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillationcited by 0
  19. 2026Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimationcited by 0
  20. 2026Grounding Large Language Models in DSGE Simulators for Policy Generation and Forecastingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
  2. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation
  3. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization
  4. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
  5. 2025AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin
  6. 2026Rethinking Groups in Critic-Free RLVR
  7. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting
  8. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning