# Leverage the Average: an Analysis of KL Regularization in RL

arXiv: https://arxiv.org/abs/2003.14089 · HTML: https://mlsanity.com/paper/2003.14089/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2020.

## Built on by (most cited first, 30 of 33)

- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Extreme Q-Learning: MaxEnt RL without Entropy](https://mlsanity.com/paper/2301.02328/) (2023)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking](https://mlsanity.com/paper/2403.03185/) (2024)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [Towards Minimax Optimality of Model-based Robust Reinforcement Learning](https://mlsanity.com/paper/2302.05372/) (2023)
- [Improved Regret Bound and Experience Replay in Regularized Policy Iteration](https://mlsanity.com/paper/2102.12611/) (2021)
- [Bayesian Bellman Operators](https://mlsanity.com/paper/2106.05012/) (2021)
- [Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.17258/) (2021)
- [Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning](https://mlsanity.com/paper/2501.14315/) (2025)
- [Latent State Marginalization as a Low-cost Approach for Improving Exploration](https://mlsanity.com/paper/2210.00999/) (2022)
- [Optimization Issues in KL-Constrained Approximate Policy Iteration](https://mlsanity.com/paper/2102.06234/) (2021)
- [Implicitly Regularized RL with Implicit Q-Values](https://mlsanity.com/paper/2108.07041/) (2021)
- [Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act](https://mlsanity.com/paper/2203.08542/) (2022)
- [Do You Need the Entropy Reward (in Practice)?](https://mlsanity.com/paper/2201.12434/) (2022)
- [Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning](https://mlsanity.com/paper/2608.00301/) (2026)
- [q-exponential family for policy optimization](https://mlsanity.com/paper/2408.07245/) (2024)
- [Dual Approximation Policy Optimization](https://mlsanity.com/paper/2410.01249/) (2024)
- [Non-Asymptotic Global Convergence of PPO-Clip](https://mlsanity.com/paper/2512.16565/) (2025)
- [Near Optimal Policy Optimization via REPS](https://mlsanity.com/paper/2103.09756/) (2021)
- [Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence](https://mlsanity.com/paper/2301.11476/) (2023)
- [Guaranteed Trust Region Optimization via Two-Phase KL Penalization](https://mlsanity.com/paper/2312.05405/) (2023)
- [On the Effect of Regularization in Policy Mirror Descent](https://mlsanity.com/paper/2507.08718/) (2025)
- [GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learning](https://mlsanity.com/paper/2603.10250/) (2026)
- [Value Mirror Descent for Reinforcement Learning](https://mlsanity.com/paper/2604.06039/) (2026)
- [Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning](https://mlsanity.com/paper/2107.05798/) (2021)
- [Error Controlled Actor-Critic](https://mlsanity.com/paper/2109.02517/) (2021)
- [Actor Loss of Soft Actor Critic Explained](https://mlsanity.com/paper/2112.15568/) (2021)
- [q-Munchausen Reinforcement Learning](https://mlsanity.com/paper/2205.07467/) (2022)
- [Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning](https://mlsanity.com/paper/2205.07885/) (2022)

## Abstract (arXiv)

Recent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance. Yet, only little is understood theoretically about why KL regularization helps, so far. We study KL regularization within an approximate value iteration scheme and show that it implicitly averages q-values. Leveraging this insight, we provide a very strong performance bound, the very first to combine two desirable aspects: a linear dependency to the horizon (instead of quadratic) and an error propagation term involving an averaging effect of the estimation errors (instead of an accumulation effect). We also study the more general case of an additional entropy regularizer. The resulting abstract scheme encompasses many existing RL algorithms. Some of our assumptions do not hold with neural networks, so we complement this theoretical analysis with an extensive empirical study.