Policy Gradient with Second Order Momentum
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
We develop Policy Gradient with Second-Order Momentum (PG-SOM), a lightweight second-order optimisation scheme for reinforcement-learning policies. PG-SOM augments the classical REINFORCE update with two exponentially weighted statistics: a first-order gradient average and a diagonal approximation of the Hessian. By preconditioning the gradient with this curvature estimate, the method adaptively rescales each parameter, yielding faster and more stable ascent of the expected return. We provide a concise derivation, establish that the diagonal Hessian estimator is unbiased and positive-definite under mild regularity assumptions, and prove that the resulting update is a descent direction in expectation. Numerical experiments on standard control benchmarks show up to a 2.1x increase in sample efficiency and a substantial reduction in variance compared to first-order and Fisher-matrix baselines. These results indicate that even coarse second-order information can deliver significant practical gains while incurring only D memory overhead for a D-parameter policy. All code and reproducibility scripts will be made publicly available.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 5 of its 15 references that are in the map. First posted on arXiv 16 May 2025.
Builds on
5 of 15 references.
References · 5
- 2015Trust Region Policy Optimizationcited by 2,580
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
- 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
- 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025A Variance-Reduced Cubic-Regularized Newton for Policy Optimization
- 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning
- 2026Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition
- 2025Second-Order Policy Gradient Methods for the Linear Quadratic Regulator
- 2025Reusing Trajectories in Policy Gradients Enables Fast Convergence
- 2025Reinforcement Learning in POMDP's via Direct Gradient Ascent
- 2024Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction
- 2026Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework