# AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control

arXiv: https://arxiv.org/abs/2406.03995 · HTML: https://mlsanity.com/paper/2406.03995/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [acados: a modular open-source framework for fast embedded optimal control](https://mlsanity.com/paper/1910.13753/) (2019)
- [Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms](https://mlsanity.com/paper/2203.07747/) (2022)
- [Mirror Learning: A Unifying Framework of Policy Optimisation](https://mlsanity.com/paper/2201.02373/) (2022)
- [Large Scale Model Predictive Control with Neural Networks and Primal Active Sets](https://mlsanity.com/paper/1910.10835/) (2019)
- [End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization](https://mlsanity.com/paper/2212.08260/) (2022)
- [Warm Start of Mixed-Integer Programs for Model Predictive Control of Hybrid Systems](https://mlsanity.com/paper/1910.08251/) (2019)
- [Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot](https://mlsanity.com/paper/2003.03200/) (2020)
- [Learning for CasADi: Data-driven Models in Numerical Optimization](https://mlsanity.com/paper/2312.05873/) (2023)
- [A Hierarchical Approach for Strategic Motion Planning in Autonomous Racing](https://mlsanity.com/paper/2212.01607/) (2022)
- [LVIS: Learning from Value Function Intervals for Contact-Aware Robot Controllers](https://mlsanity.com/paper/1809.05802/) (2018)
- [OffCon3: What is state of the art anyway?](https://mlsanity.com/paper/2101.11331/) (2021)
- [Stability-Constrained Markov Decision Processes Using MPC](https://mlsanity.com/paper/2102.01383/) (2021)
- [Predictive Control with Learning-Based Terminal Costs Using Approximate Value Iteration](https://mlsanity.com/paper/2212.00361/) (2022)
- [Imitation Learning from Nonlinear MPC via the Exact Q-Loss and its Gauss-Newton Approximation](https://mlsanity.com/paper/2304.01782/) (2023)
- [Approximate infinite-horizon predictive control](https://mlsanity.com/paper/2111.08319/) (2021)
- [Frenet-Cartesian Model Representations for Automotive Obstacle Avoidance within Nonlinear MPC](https://mlsanity.com/paper/2212.13115/) (2022)
- [Reinforcement Learning and Distributed Model Predictive Control for Conflict Resolution in Highly Constrained Spaces](https://mlsanity.com/paper/2302.01586/) (2023)
- [Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gap](https://mlsanity.com/paper/2306.11271/) (2023)

## Built on by (most cited first, 6 of 6)

- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Differentiable Nonlinear Model Predictive Control](https://mlsanity.com/paper/2505.01353/) (2025)
- [Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)](https://mlsanity.com/paper/2508.16474/) (2025)
- [CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization](https://mlsanity.com/paper/2602.19699/) (2026)
- [Autonomous Wheel Loader Navigation Using Goal-Conditioned Actor-Critic MPC](https://mlsanity.com/paper/2409.15717/) (2024)
- [Q-Guided Stein Variational Model Predictive Control via RL-informed Policy Prior](https://mlsanity.com/paper/2507.06625/) (2025)

## Abstract (arXiv)

AcMPC and acRL are two powerful control strategies with, arguably, complementary advantages. In this work, we show how actor-critic acRL techniques can be leveraged to improve the performance of acMPC. The acRL critic is used as an approximation of the optimal value function, and an actor roll-out provides an initial guess for primal variables of the acMPC. A parallel control architecture is proposed where each acMPC instance is solved twice for different initial guesses. Besides the actor roll-out initialization, a shifted initialization from the previous solution is used. Thereafter, the actor and the critic are again used to approximately evaluate the infinite horizon cost of these trajectories. The control actions from the lowest-cost trajectory are applied to the system at each time step. We establish that the proposed algorithm is guaranteed to outperform the original acRL policy plus an error term that depends on the accuracy of the critic and decays with the horizon length of the acMPC formulation. Moreover, we do not require globally optimal solutions for these guarantees to hold. The approach is demonstrated on an illustrative toy example and an acAD overtaking scenario.