mlsanityData to 7 Oct 2026

Leverage the Average: an Analysis of KL Regularization in RL

arXiv 2003.14089 ↗First posted 31 Mar 2020cs.LG stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

Recent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance. Yet, only little is understood theoretically about why KL regularization helps, so far. We study KL regularization within an approximate value iteration scheme and show that it implicitly averages q-values. Leveraging this insight, we provide a very strong performance bound, the very first to combine two desirable aspects: a linear dependency to the horizon (instead of quadratic) and an error propagation term involving an averaging effect of the estimation errors (instead of an accumulation effect). We also study the more general case of an additional entropy regularizer. The resulting abstract scheme encompasses many existing RL algorithms. Some of our assumptions do not hold with neural networks, so we complement this theoretical analysis with an extensive empirical study.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 5; 2023: 5; 2024: 3; 2025: 4; 2026: 4.

33 papers.

Sort:

By area:

  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  2. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  3. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  4. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  5. 2021Adversarially Guided Actor-Criticcited by 47
  6. 2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 21
  7. 2021Improved Regret Bound and Experience Replay in Regularized Policy Iterationcited by 18
  8. 2021Bayesian Bellman Operatorscited by 16
  9. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  10. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  11. 2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learningcited by 12
  12. 2021Optimization Issues in KL-Constrained Approximate Policy Iterationcited by 10
21 more
  1. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
  2. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  3. 2022Do You Need the Entropy Reward (in Practice)?cited by 6
  4. 2024q-exponential family for policy optimizationcited by 2
  5. 2025Non-Asymptotic Global Convergence of PPO-Clipcited by 2
  6. 2021Near Optimal Policy Optimization via REPScited by 1
  7. 2023Generalized Munchausen Reinforcement Learning using Tsallis KL Divergencecited by 1
  8. 2023Guaranteed Trust Region Optimization via Two-Phase KL Penalizationcited by 1
  9. 2024Dual Approximation Policy Optimizationcited by 1
  10. 2025On the Effect of Regularization in Policy Mirror Descentcited by 1
  11. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  12. 2026Value Mirror Descent for Reinforcement Learningcited by 1
  13. 2026Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learningcited by 1
  14. 2021Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learningcited by 0
  15. 2021Error Controlled Actor-Criticcited by 0
  16. 2021Actor Loss of Soft Actor Critic Explainedcited by 0
  17. 2022q-Munchausen Reinforcement Learningcited by 0
  18. 2022Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learningcited by 0
  19. 2023Acceleration in Policy Optimizationcited by 0
  20. 2025Symmetric Behavior Regularized Policy Optimizationcited by 0
  21. 2026Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitionerscited by 0