mlsanityData to 7 Oct 2026

Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences

arXiv 2107.08285 ↗First posted 17 Jul 2021cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Approximate Policy Iteration (API) algorithms alternate between (approximate) policy evaluation and (approximate) greedification. Many different approaches have been explored for approximate policy evaluation, but less is understood about approximate greedification and what choices guarantee policy improvement. In this work, we investigate approximate greedification when reducing the KL divergence between the parameterized policy and the Boltzmann distribution over action values. In particular, we investigate the difference between the forward and reverse KL divergences, with varying degrees of entropy regularization. We show that the reverse KL has stronger policy improvement guarantees, but that reducing the forward KL can result in a worse policy. We also demonstrate, however, that a large enough reduction of the forward KL can induce improvement under additional assumptions. Empirically, we show on simple continuous-action environments that the forward KL can induce more exploration, but at the cost of a more suboptimal policy. No significant differences were observed in the discrete-action setting or on a suite of benchmark problems. Throughout, we highlight that many policy gradient methods can be seen as an instance of API, with either the forward or reverse KL for the policy update, and discuss next steps for understanding and improving our policy optimization algorithms.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 44 of its 87 references that are in the map. First posted on arXiv 17 Jul 2021.

Builds on

44 of 87 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  12. 2016Input Convex Neural Networkscited by 407
32 more
  1. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  2. 2018Maximum a Posteriori Policy Optimisationcited by 318
  3. 2019A Theory of Regularized Markov Decision Processescited by 279
  4. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  5. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  7. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  8. 2017A unified view of entropy-regularized Markov decision processescited by 174
  9. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  10. 2018Understanding the impact of entropy on policy optimizationcited by 171
  11. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  12. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  13. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  14. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  15. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  16. 2017Time Limits in Reinforcement Learningcited by 72
  17. 2019Is the Policy Gradient a Gradient?cited by 40
  18. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
  19. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  20. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 37
  21. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  22. 2019The Value Function Polytope in Reinforcement Learningcited by 29
  23. 2021Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergencecited by 29
  24. 2016Combining policy gradient and Q-learningcited by 23
  25. 2019CAQL: Continuous Action Q-Learningcited by 22
  26. 2019MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experimentscited by 17
  27. 2020An operator view of policy gradient methodscited by 17
  28. 2019Entropic Regularization of Markov Decision Processescited by 15
  29. 2016Improving Policy Gradient by Exploring Under-appreciated Rewardscited by 14
  30. 2019Deep Conservative Policy Iterationcited by 14
  31. 2014Classification-based Approximate Policy Iteration: Experiments and Extended Discussionscited by 3
  32. 2020Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 11; 2025: 11; 2026: 10.

40 papers.

Sort:

By area:

  1. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  2. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  3. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60
  4. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  5. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
  6. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  7. 2022Critic Sequential Monte Carlocited by 16
  8. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 16
  9. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Gamescited by 13
  10. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  11. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  12. 2024Variational Schr"odinger Diffusion Modelscited by 9
28 more
  1. 2022HesScale: Scalable Computation of Hessian Diagonalscited by 8
  2. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8
  3. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spacescited by 7
  4. 2022The Geometry of Robust Value Functionscited by 6
  5. 2024AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learningcited by 6
  6. 2024Value Improved Actor Critic Algorithmscited by 6
  7. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  8. 2025Trust-Region Twisted Policy Improvementcited by 5
  9. 2026Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillationcited by 5
  10. 2022TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasetscited by 4
  11. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
  12. 2025Twice Sequential Monte Carlo for Tree Searchcited by 3
  13. 2025Diffusion Alignment as Variational Expectation-Maximizationcited by 2
  14. 2023Generalized Munchausen Reinforcement Learning using Tsallis KL Divergencecited by 1
  15. 2024Policy Bifurcation in Safe Reinforcement Learningcited by 1
  16. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learningcited by 1
  17. 2025Lifelong Learning with Behavior Consolidation for Vehicle Routingcited by 1
  18. 2025Coupled Variational Reinforcement Learning for Language Model General Reasoningcited by 1
  19. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  20. 2026Consistently Informative Soft-Label Temperature for Knowledge Distillationcited by 1
  21. 2024Shared learning of powertrain control policies for vehicle fleetscited by 0
  22. 2025Symmetric Behavior Regularized Policy Optimizationcited by 0
  23. 2026Proximal Policy Optimization for Amortized Discrete Samplingcited by 0
  24. 2026Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetrycited by 0
  25. 2026Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitionerscited by 0
  26. 2026Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Qualitycited by 0
  27. 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectivescited by 0
  28. 2026Towards Optimal Policy Improvementcited by 0