mlsanityData to 7 Oct 2026

A Unified Framework for Rethinking Policy Divergence Measures in GRPO

arXiv 2602.05494 ↗First posted 5 Feb 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verified Reward (RLVR) has emerged as a critical paradigm for advancing the reasoning capabilities of Large Language Models (LLMs). Most existing RLVR methods, such as GRPO and its variants, ensure stable updates by constraining policy divergence through clipping likelihood ratios. This paper introduces a unified clipping framework that characterizes existing methods via a general notion of policy divergence, encompassing both likelihood ratios and Kullback-Leibler (KL) divergences and extending to alternative measures. The framework provides a principled foundation for systematically analyzing how different policy divergence measures affect exploration and performance. We further identify the KL3 estimator, a variance-reduced Monte Carlo estimator of the KL divergence, as a key policy divergence constraint. We theoretically demonstrate that the KL3-based constraint is mathematically equivalent to an asymmetric ratio-based clipping that reallocates probability mass toward high-confidence actions, promoting stronger exploration while retaining the simplicity of GRPO-style methods. Empirical results on mathematical reasoning benchmarks demonstrate that incorporating the KL3 estimator into GRPO improves both training stability and final performance, highlighting the importance of principled policy divergence constraints in policy optimization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 42 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

31 of 42 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMs26 Dec
  2. 2025Soft Adaptive Policy Optimization25 Nov
  3. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models30 Sep
  4. 2025DCPO: Dynamic Clipping Policy Optimization2 Sep
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning23 May
  7. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  12. 2025Group Sequence Policy Optimizationcited by 788
7 more
  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  3. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  4. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  5. 2019Truly Proximal Policy Optimizationcited by 68
  6. 2019Trust Region-Guided Proximal Policy Optimizationcited by 36
  7. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimizationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  2. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  4. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  5. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
  6. 2025SSPO: Subsentence-level Policy Optimization
  7. 2026GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings
  8. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping