mlsanityData to 14 Oct 2026

Policy Gradient Algorithms Implicitly Optimize by Continuation

arXiv 2305.06851 ↗First posted 11 May 2023cs.LG math.OC stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Direct policy optimization in reinforcement learning is usually solved with policy-gradient algorithms, which optimize policy parameters via stochastic gradient ascent. This paper provides a new theoretical interpretation and justification of these algorithms. First, we formulate direct policy optimization in the optimization by continuation framework. The latter is a framework for optimizing nonconvex functions where a sequence of surrogate objective functions, called continuations, are locally optimized. Second, we show that optimizing affine Gaussian policies and performing entropy regularization can be interpreted as implicitly optimizing deterministic policies by continuation. Based on these theoretical results, we argue that exploration in policy-gradient algorithms consists in computing a continuation of the return of the policy at hand, and that the variance of policies should be history-dependent functions adapted to avoid local extrema rather than to maximize the return of the policy.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 32 of its 63 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

32 of 63 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  9. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  10. 2018Understanding the impact of entropy on policy optimizationcited by 176
  11. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  12. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
20 more
  1. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  2. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  3. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  4. 2018Evolved Policy Gradientscited by 88
  5. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  6. 2017Towards Generalization and Simplicity in Continuous Controlcited by 68
  7. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  8. 2015On Graduated Optimization for Stochastic Non-Convex Problemscited by 34
  9. 2021Geometric Entropic Explorationcited by 32
  10. 2021Regularized Policies are Reward Robustcited by 32
  11. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  12. 2018Clipped Action Policy Gradientcited by 19
  13. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  14. 2016Improving Policy Gradient by Exploring Under-appreciated Rewardscited by 14
  15. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  16. 2022On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spacescited by 11
  17. 2020Policy Gradient using Weak Derivatives for Reinforcement Learningcited by 7
  18. 2020A Novel Evolution Strategy with Directional Gaussian Smoothing for Blackbox Optimizationcited by 6
  19. 2022On learning history based policies for controlling Markov decision processescited by 5
  20. 2019Graduated Optimization of Black-Box Functionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradientscited by 12
  2. 2024Behind the Myth of Exploration in Policy Gradientscited by 3
  3. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  4. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0