mlsanityData to 14 Oct 2026

On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning

arXiv 2505.17508 ↗First posted 23 May 2025cs.LG cs.AI cs.CLcited by 30

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs). KL regularization is ubiquitous, yet the design surface, choice of KL direction (forward vs. reverse), normalization (normalized vs. unnormalized), and estimator (k1/k2/k3), is scattered across the literature and often intertwined with off-policy estimation. We ask a focused question: under the off-policy setting, what weighting is required for each KL variant so that the surrogate we optimize yields the exact gradient of the intended KL-regularized objective? We answer this with a compact, unified derivation we call the Regularized Policy Gradient (RPG) view. RPG (i) unifies normalized and unnormalized KL variants and shows that the widely-used k3 penalty is exactly the unnormalized KL; (ii) specifies conditions under which REINFORCE-style losses with stop-gradient are gradient-equivalent to fully differentiable surrogates; (iii) identifies and corrects an off-policy importance-weighting mismatch in GRPO's KL term; and (iv) introduces RPG-Style Clip, a clipped-importance-sampling step within RPG-REINFORCE that enables stable, off-policy policy-gradient training at scale. On mathematical reasoning benchmarks (AIME24, AIME25), RPG-REINFORCE with RPG-Style Clip improves accuracy by up to +6 absolute percentage points over DAPO. We extend our experiments to 8K context length, and RPG-REINFORCE with RPG-Style Clip achieves 52% accuracy on AIME25, surpassing the official Qwen3-4B-Instruct model (47%). Notably, RPG is a stable and scalable RL algorithm for LLM reasoning, realized via (a) a KL-correct objective, (b) clipped importance sampling, and (c) an iterative reference-policy update scheme. Project Page: https://github.com/complex-reasoning/RPG.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 50 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

19 of 50 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  7. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  8. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization11 Oct
  9. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 9

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  8. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  9. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 22.

30 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  3. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 29
  4. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  5. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  6. 2025Proximal Diffusion Neural Samplercited by 16
  7. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 12
  8. 2026Hybrid Policy Distillation for LLMscited by 9
  9. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 8
  10. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillationcited by 8
  11. 2026Self-Distilled Policy Gradientcited by 8
  12. 2025Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuningcited by 7
18 more
  1. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  2. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuningcited by 6
  3. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 5
  4. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  5. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 3
  6. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMscited by 3
  7. 2026Scaling Reinforcement Learning for Diffusion Models via Velocity Matchingcited by 3
  8. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 2
  9. 2026Verifiable Rewards for Calibrated Probabilistic Forecastingcited by 2
  10. 2025Dataset Construction for Training LLM to Learn Analog Circuit Knowledgecited by 1
  11. 2025Why Alignment Must Precede Distillation: A Minimal Working Explanationcited by 1
  12. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimisercited by 1
  13. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPOcited by 0
  14. 2026Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusioncited by 0
  15. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0
  16. 2026KLip-PPO: A per-sample KL perspective on PPO-Clipcited by 0
  17. 2026UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemmacited by 0
  18. 2026Bellman Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  2. 2026KL for a KL: On-Policy Distillation with Control Variate Baseline
  3. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  4. 2026Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
  5. 2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning
  6. 2026On KL-Regularized Policy Optimization
  7. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning
  8. 2026LLMs Can Learn to Reason Via Off-Policy RL