mlsanityData to 14 Oct 2026

On a few pitfalls in KL divergence gradient estimation for RL

arXiv 2506.09477 ↗First posted 11 Jun 2025cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

We point out a few pitfalls in implementing gradient estimation for KL divergence in RL training for LLM, as seen in a number of open source projects and papers. The first major pitfall is to differentiate through the KL estimate as loss functions to minimize KL divergence. We show that such implementations are generally incorrect and do not produce the desired KL gradient. Secondly, we show that some implementations do not account for the sequential nature of the estimation problem and produce a partial gradient at best. We demonstrate the impact of such issues with illustrative tabular and LLM experiments, and show the correct way to implement the KL gradient.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

26 of 37 references.

References · 26

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2015Trust Region Policy Optimizationcited by 2,580
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
14 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  4. 2013Black Box Variational Inferencecited by 319
  5. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  6. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  7. 2015Gradient Estimation Using Stochastic Computation Graphscited by 121
  8. 2016Variational inference for Monte Carlo objectivescited by 84
  9. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  10. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  11. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  12. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  13. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12
  14. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithmcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 22.

29 papers.

Sort:

By area:

  1. 2026Self-Distillation Enables Continual Learningcited by 247
  2. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  3. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  4. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  5. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 32
  6. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  7. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 26
  8. 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 19
  9. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  10. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillationcited by 8
  11. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 6
  12. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoningcited by 5
17 more
  1. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 5
  2. 2026f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Datacited by 4
  3. 2025Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspectivecited by 3
  4. 2026Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrabilitycited by 3
  5. 2026Scaling Reward Modeling without Human Supervisioncited by 1
  6. 2026Sample Efficient Generative Molecular Optimization with Joint Self-Improvementcited by 0
  7. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0
  8. 2026Beyond Distribution Sharpening: The Importance of Task Rewardscited by 0
  9. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  10. 2026Interpreting Language Model Hidden States at Scalecited by 0
  11. 2026Suan: Rectifying Direct Preference Safety Alignment in Large Language Modelscited by 0
  12. 2026Scaling Properties of Same-Family On-Policy Distillationcited by 0
  13. 2026On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamicscited by 0
  14. 2026The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learningcited by 0
  15. 2026Ontology Concept Overlap as a Training Signal: Knowledge-Grounded Reinforcement Learning for Clinical Question Answeringcited by 0
  16. 2026Reward-Driven Learning under Prompt-Level Differential Privacycited by 0
  17. 2026When KL Regularization Misfires in Group Policy Optimizationcited by 0