mlsanityData to 14 Oct 2026

Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery

arXiv 2609.19164 ↗First posted 27 Aug 2026cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

In verifier-style RLVR, group-relative optimization often treats advantage scale as an implementation detail. This paper separates two low-variance cases: sub-resolution jitter that should not become a preference signal, and credible but small cardinal gaps that should be learned without distorting KL calibration. We propose an advantage-scale three-way calibration interface: the same within-group scale denominator simultaneously determines the reward-branch strength, prompt-level batch weight, and the effective KL calibration induced when the reward branch is re-expressed on the original cardinal scale. This interface explains why RLOO / Dr.GRPO can let credible small gaps become KL dominated, whereas GRPO's standard-deviation denominator can amplify tiny gaps without bound. Based on this interface, we further introduce the Reward-Resolution Protocol and MaxNorm-AC, respectively filtering sub-resolution gaps and providing bounded cardinal recovery on credible nonzero gaps. Across dense / MoE architectures and math / code reasoning, MaxNorm-AC improves over the strongest robust-scale baseline while truncating the low-variance inverse-scale tail.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

47 of 55 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  6. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  7. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 241
  8. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  9. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  10. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  11. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 34
  12. 2026Your Group-Relative Advantage Is Biasedcited by 34
35 more
  1. 2025NGRPO: Negative-enhanced Group Relative Policy Optimizationcited by 32
  2. 2025Promoting Efficient Reasoning with Verifiable Stepwise Rewardcited by 28
  3. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  4. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  5. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  6. 2025Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgetscited by 17
  7. 2025Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewardscited by 16
  8. 2025Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimizationcited by 15
  9. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimizationcited by 14
  10. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 13
  11. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoningcited by 12
  12. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 12
  13. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoningcited by 12
  14. 2026RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agentscited by 11
  15. 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalizationcited by 11
  16. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferencescited by 10
  17. 2026Latent-GRPO: Group Relative Policy Optimization for Latent Reasoningcited by 10
  18. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  19. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learningcited by 6
  20. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradientscited by 5
  21. 2026GOPO: Policy Optimization using Ranked Rewardscited by 5
  22. 2026Calibration-Aware Policy Optimization for Reasoning LLMscited by 5
  23. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoningcited by 5
  24. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shapingcited by 4
  25. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  26. 2026On the Hidden Objective Biases of Group-based Reinforcement Learningcited by 3
  27. 2026RVPO: Risk-Sensitive Alignment via Variance Regularizationcited by 3
  28. 2026Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplexcited by 3
  29. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  30. 2026On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapsecited by 3
  31. 2026Quantifying Empirical Compute-Supervision Tradeoffs in RLVRcited by 2
  32. 2026MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Followingcited by 2
  33. 2026Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewardscited by 1
  34. 2026EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidancecited by 1
  35. 2026BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation
  2. 2026Skip-Connected Policy Optimization for Implicit Advantage
  3. 2026When KL Regularization Misfires in Group Policy Optimization
  4. 2026Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards
  5. 2025GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNA
  6. 2026On Advantage Estimates for Max@K Policy Gradients
  7. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
  8. 2026STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking