mlsanityData to 14 Oct 2026

When KL Regularization Misfires in Group Policy Optimization

arXiv 2610.12161 ↗First posted 8 Oct 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

17 of 27 references.

References · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  3. 2025Group Sequence Policy Optimizationcited by 835
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  5. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 96
  6. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 29
  7. 2025What is the Alignment Objective of GRPO?cited by 23
  8. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  9. 2026GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignmentcited by 6
  10. 2026Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectificationcited by 5
  11. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 3
  12. 2026sGPO: Trading Inference FLOPs for Training Efficiency in RLVRcited by 3
5 more
  1. 2026Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimizationcited by 3
  2. 2026AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JDcited by 2
  3. 2026Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groupscited by 2
  4. 2026Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Spacecited by 1
  5. 2026AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedbackcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
  2. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  3. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  4. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
  5. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
  6. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularization
  7. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapse
  8. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients