mlsanityData to 7 Oct 2026

A Comedy of Estimators: On KL Regularization in RL Training of LLMs

arXiv 2512.21852 ↗First posted 26 Dec 2025cs.LG cs.AIcited by 18

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The reasoning performance of large language models (LLMs) can be substantially improved by training them with reinforcement learning (RL). The RL objective for LLM training involves a regularization term, which is the reverse Kullback-Leibler (KL) divergence between the trained policy and the reference policy. Since computing the KL divergence exactly is intractable, various estimators are used in practice to estimate it from on-policy samples. Despite its wide adoption, including in several open-source libraries, there is no systematic study analyzing the numerous ways of incorporating KL estimators in the objective and their effect on the downstream performance of RL-trained models. Recent works show that prevailing practices for incorporating KL regularization do not provide correct gradients for stated objectives, creating a discrepancy between the objective and its implementation. In this paper, we further analyze these practices and study the gradients of several estimators configurations, revealing how design choices shape gradient bias. We substantiate these findings with empirical observations by RL fine-tuning textttQwen2.5-7B, textttLlama-3.1-8B-Instruct and textttQwen3-4B-Instruct-2507 with different configurations and evaluating their performance on both in- and out-of-distribution tasks. Through our analysis, we observe that, in on-policy settings: (1) estimator configurations with biased gradients can result in training instabilities; and (2) using estimator configurations resulting in unbiased gradients leads to better performance on in-domain as well as out-of-domain tasks. We also investigate the performance resulting from different KL configurations in off-policy settings and observe that KL regularization can help stabilize off-policy RL training resulting from asynchronous setups.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 60 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

43 of 60 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning23 May
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning10 Feb
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  8. 2023Let's Verify Step by Step31 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2024Qwen2.5 Technical Reportcited by 4,713
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
22 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2025Group Sequence Policy Optimizationcited by 788
  8. 2013Black Box Variational Inferencecited by 317
  9. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  10. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  11. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  12. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  13. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  14. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  15. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  16. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60
  17. 2025Reinforcement Learning from Human Feedbackcited by 53
  18. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  19. 2024AI-Assisted Generation of Difficult Math Questionscited by 37
  20. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  21. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  22. 2025Better Estimation of the Kullback--Leibler Divergence Between Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area:

  1. 2026Privileged Information Distillation for Language Modelscited by 79
  2. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 7
  3. 2026Hybrid Policy Distillation for LLMscited by 7
  4. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillationcited by 7
  5. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  6. 2026Continuous Latent Contexts Enable Efficient Online Learning in Transformerscited by 3
  7. 2026Tail-Aware Top-k On-Policy Distillationcited by 3
  8. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 2
  9. 2026Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategiescited by 2
  10. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimisercited by 1
  11. 2026Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrabilitycited by 1
  12. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPOcited by 0
6 more
  1. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisationcited by 0
  2. 2026Beyond Distribution Sharpening: The Importance of Task Rewardscited by 0
  3. 2026LightAVSeg: Lightweight Audio-Visual Segmentationcited by 0
  4. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  5. 2026One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agentscited by 0
  6. 2026When KL Regularization Misfires in Group Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  2. 2026On KL-Regularized Policy Optimization
  3. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
  4. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  5. 2025APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs
  7. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  8. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity