# Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery

arXiv: https://arxiv.org/abs/2609.19164 · HTML: https://mlsanity.com/paper/2609.19164/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 55 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 30 of 47)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization](https://mlsanity.com/paper/2601.05242/) (2026)
- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles](https://mlsanity.com/paper/2401.00243/) (2023)
- [Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training](https://mlsanity.com/paper/2505.22257/) (2025)
- [Your Group-Relative Advantage Is Biased](https://mlsanity.com/paper/2601.08521/) (2026)
- [NGRPO: Negative-enhanced Group Relative Policy Optimization](https://mlsanity.com/paper/2509.18851/) (2025)
- [Promoting Efficient Reasoning with Verifiable Stepwise Reward](https://mlsanity.com/paper/2508.10293/) (2025)
- [Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach](https://mlsanity.com/paper/2503.21819/) (2025)
- [Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic](https://mlsanity.com/paper/2603.01162/) (2026)
- [Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO](https://mlsanity.com/paper/2505.11595/) (2025)
- [Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets](https://mlsanity.com/paper/2508.14094/) (2025)
- [Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2511.03710/) (2025)
- [Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization](https://mlsanity.com/paper/2502.01652/) (2025)
- [Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization](https://mlsanity.com/paper/2510.01555/) (2025)
- [Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients](https://mlsanity.com/paper/2510.18924/) (2025)
- [Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning](https://mlsanity.com/paper/2509.09284/) (2025)
- [Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularization](https://mlsanity.com/paper/2601.10201/) (2026)
- [WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning](https://mlsanity.com/paper/2602.17025/) (2026)
- [RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents](https://mlsanity.com/paper/2602.03025/) (2026)
- [PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization](https://mlsanity.com/paper/2603.26535/) (2026)
- [λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences](https://mlsanity.com/paper/2510.06870/) (2025)
- [Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning](https://mlsanity.com/paper/2604.27998/) (2026)
- [Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation](https://mlsanity.com/paper/2605.21125/) (2026)

## Built on by (most cited first, 1 of 1)

- [Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders](https://mlsanity.com/paper/2610.17771/) (2026)

## Abstract (arXiv)

In verifier-style RLVR, group-relative optimization often treats advantage scale as an implementation detail. This paper separates two low-variance cases: sub-resolution jitter that should not become a preference signal, and credible but small cardinal gaps that should be learned without distorting KL calibration. We propose an advantage-scale three-way calibration interface: the same within-group scale denominator simultaneously determines the reward-branch strength, prompt-level batch weight, and the effective KL calibration induced when the reward branch is re-expressed on the original cardinal scale. This interface explains why RLOO / Dr.GRPO can let credible small gaps become KL dominated, whereas GRPO's standard-deviation denominator can amplify tiny gaps without bound. Based on this interface, we further introduce the Reward-Resolution Protocol and MaxNorm-AC, respectively filtering sub-resolution gaps and providing bounded cardinal recovery on credible nonzero gaps. Across dense / MoE architectures and math / code reasoning, MaxNorm-AC improves over the strongest robust-scale baseline while truncating the low-variance inverse-scale tail.