mlsanityData to 7 Oct 2026

Momentum-Based Policy Gradient with Second-Order Information

arXiv 2205.08253 ↗First posted 17 May 2022cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Variance-reduced gradient estimators for policy gradient methods have been one of the main focus of research in the reinforcement learning in recent years as they allow acceleration of the estimation process. We propose a variance-reduced policy-gradient method, called SHARP, which incorporates second-order information into stochastic gradient descent (SGD) using momentum with a time-varying learning rate. SHARP algorithm is parameter-free, achieving ε-approximate first-order stationary point with O(ε-3) number of trajectories, while using a batch size of O(1) at each iteration. Unlike most previous work, our proposed algorithm does not require importance sampling which can compromise the advantage of variance reduction process. Moreover, the variance of estimation error decays with the fast rate of O(1/t2/3) where t is the number of iterations. Our extensive experimental evaluations show the effectiveness of the proposed algorithm on various control tasks and its advantage over the state of the art in practice.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 17 May 2022.

Builds on

28 of 46 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  5. 2017Deep Reinforcement Learning that Matterscited by 727
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  7. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  8. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  9. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  10. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  11. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 116
  12. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
16 more
  1. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  2. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  3. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 58
  4. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 58
  5. 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 39
  6. 2019One Sample Stochastic Frank-Wolfecited by 32
  7. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  8. 2020Momentum-Based Policy Gradient Methodscited by 28
  9. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 23
  10. 2020Stochastic Recursive Momentum for Policy Gradient Methodscited by 22
  11. 2019Policy Optimization with Stochastic Mirror Descentcited by 17
  12. 2020A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learningcited by 17
  13. 2021Bregman Gradient Policy Optimizationcited by 15
  14. 2022PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimationcited by 15
  15. 2021Better SGD using Second-order Momentumcited by 14
  16. 2021STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 2; 2026: 1.

11 papers.

  1. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  2. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 24
  3. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  4. 2024Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPscited by 11
  5. 2024Sample-Efficient Constrained Reinforcement Learning with General Parameterizationcited by 11
  6. 2024Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reductioncited by 4
  7. 2025Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limitscited by 3
  8. 2024On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimizationcited by 1
  9. 2026RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimizationcited by 1
  10. 2024Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learningcited by 0
  11. 2025Improved Convergence in Parameter-Agnostic Error Feedback through Momentumcited by 0