# Quasi-Newton Iteration in Deterministic Policy Gradient

arXiv: https://arxiv.org/abs/2203.13854 · HTML: https://mlsanity.com/paper/2203.13854/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 18 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 4 of 4)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [An Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/1811.12560/) (2018)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Combining system identification with reinforcement learning-based MPC](https://mlsanity.com/paper/2004.03265/) (2020)

## Built on by (most cited first, 3 of 3)

- [Reinforced Model Predictive Control via Trust-Region Quasi-Newton Policy Optimization](https://mlsanity.com/paper/2405.17983/) (2024)
- [Computationally efficient Gauss-Newton reinforcement learning for model predictive control](https://mlsanity.com/paper/2508.02441/) (2025)
- [Second-Order Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/2511.02095/) (2025)

## Abstract (arXiv)

This paper presents a model-free approximation for the Hessian of the performance of deterministic policies to use in the context of Reinforcement Learning based on Quasi-Newton steps in the policy parameters. We show that the approximate Hessian converges to the exact Hessian at the optimal policy, and allows for a superlinear convergence in the learning, provided that the policy parametrization is rich. The natural policy gradient method can be interpreted as a particular case of the proposed method. We analytically verify the formulation in a simple linear case and compare the convergence of the proposed method with the natural policy gradient in a nonlinear example.