mlsanityData to 7 Oct 2026

Stochastic Dimension-reduced Second-order Methods for Policy Optimization

arXiv 2301.12174 ↗First posted 28 Jan 2023math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose several new stochastic second-order algorithms for policy optimization that only require gradient and Hessian-vector product in each iteration, making them computationally efficient and comparable to policy gradient methods. Specifically, we propose a dimension-reduced second-order method (DR-SOPO) which repeatedly solves a projected two-dimensional trust region subproblem. We show that DR-SOPO obtains an O(ε-3.5) complexity for reaching approximate first-order stationary condition and certain subspace second-order stationary condition. In addition, we present an enhanced algorithm (DVR-SOPO) which further improves the complexity to O(ε-3) based on the variance reduction technique. Preliminary experiments show that our proposed algorithms perform favorably compared with stochastic and variance-reduced policy gradient methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

31 of 58 references.

References · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  7. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  8. 2017How to Escape Saddle Points Efficientlycited by 284
  9. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  10. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  11. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  12. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
19 more
  1. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  2. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  3. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  4. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  5. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  6. 2018Escaping Saddles with Stochastic Gradientscited by 66
  7. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  8. 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
  9. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  10. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  11. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 58
  12. 2017Stochastic Cubic Regularization for Fast Nonconvex Optimizationcited by 55
  13. 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 39
  14. 2020Momentum-Based Policy Gradient Methodscited by 28
  15. 2020Stochastic Recursive Momentum for Policy Gradient Methodscited by 22
  16. 2020A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learningcited by 17
  17. 2020Sample Complexity of Policy Gradient Finding Second-Order Stationary Pointscited by 16
  18. 2019A Fully Stochastic Second-Order Trust Region Methodcited by 8
  19. 2019Quasi-Newton Trust Region Policy Optimizationcited by 7

Built on by

Not cited yet.