# Computationally efficient Gauss-Newton reinforcement learning for model predictive control

arXiv: https://arxiv.org/abs/2508.02441 · HTML: https://mlsanity.com/paper/2508.02441/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Exact and Inexact Subsampled Newton Methods for Optimization](https://mlsanity.com/paper/1609.08502/) (2016)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Quasi-Newton Trust Region Policy Optimization](https://mlsanity.com/paper/1912.11912/) (2019)
- [Modern Machine Learning Tools for Monitoring and Control of Industrial Processes: A Survey](https://mlsanity.com/paper/2209.11123/) (2022)
- [End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control](https://mlsanity.com/paper/2308.01674/) (2023)
- [Reinforced Model Predictive Control via Trust-Region Quasi-Newton Policy Optimization](https://mlsanity.com/paper/2405.17983/) (2024)
- [Quasi-Newton Iteration in Deterministic Policy Gradient](https://mlsanity.com/paper/2203.13854/) (2022)
- [Stochastic Trust Region Inexact Newton Method for Large-scale Machine Learning](https://mlsanity.com/paper/1812.10426/) (2018)

## Abstract (arXiv)

Model predictive control (MPC) is widely used in process control due to its interpretability and ability to handle constraints. As a parametric policy in reinforcement learning (RL), MPC offers strong initial performance and low data requirements compared to black-box policies like neural networks. However, most RL methods rely on first-order updates, which scale well to large parameter spaces but converge at most linearly, making them inefficient when each policy update requires solving an optimal control problem, as is the case with MPC. While MPC policies are typically low parameterized and thus amenable to second-order approaches, existing second-order methods demand second-order policy derivatives, which can be computationally intractable. This work introduces a Gauss-Newton approximation of the deterministic policy Hessian that eliminates the need for second-order policy derivatives, enabling superlinear convergence with minimal computational overhead. To further improve robustness, we propose a momentum-based Hessian averaging scheme for stable training under noisy estimates coupled with an adaptive trustregion. We demonstrate the effectiveness of the approach on a nonlinear continuously stirred tank reactor (CSTR), showing faster convergence and improved data efficiency over state-of-the-art firstorder methods and deep RL approaches.