# Quasi-Newton Trust Region Policy Optimization

arXiv: https://arxiv.org/abs/1912.11912 · HTML: https://mlsanity.com/paper/1912.11912/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Dec 2019.

## Built on by (most cited first, 7 of 7)

- [Reinforced Model Predictive Control via Trust-Region Quasi-Newton Policy Optimization](https://mlsanity.com/paper/2405.17983/) (2024)
- [Policy Newton Algorithm in Reproducing Kernel Hilbert Space](https://mlsanity.com/paper/2506.01597/) (2025)
- [EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization](https://mlsanity.com/paper/2110.13373/) (2021)
- [Entropy Augmented Reinforcement Learning](https://mlsanity.com/paper/2208.09322/) (2022)
- [Stochastic Dimension-reduced Second-order Methods for Policy Optimization](https://mlsanity.com/paper/2301.12174/) (2023)
- [Computationally efficient Gauss-Newton reinforcement learning for model predictive control](https://mlsanity.com/paper/2508.02441/) (2025)
- [Ratio-Variance Regularized Policy Optimization](https://mlsanity.com/paper/2605.26784/) (2026)

## Abstract (arXiv)

We propose a trust region method for policy optimization that employs Quasi-Newton approximation for the Hessian, called Quasi-Newton Trust Region Policy Optimization QNTRPO. Gradient descent is the de facto algorithm for reinforcement learning tasks with continuous controls. The algorithm has achieved state-of-the-art performance when used in reinforcement learning across a wide range of tasks. However, the algorithm suffers from a number of drawbacks including: lack of stepsize selection criterion, and slow convergence. We investigate the use of a trust region method using dogleg step and a Quasi-Newton approximation for the Hessian for policy optimization. We demonstrate through numerical experiments over a wide range of challenging continuous control tasks that our particular choice is efficient in terms of number of samples and improves performance