# Model Predictive Control and Reinforcement Learning: A Unified Framework Based on Dynamic Programming

arXiv: https://arxiv.org/abs/2406.00592 · HTML: https://mlsanity.com/paper/2406.00592/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 115 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 21 of 21)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Active Learning for Nonlinear System Identification with Guarantees](https://mlsanity.com/paper/2006.10277/) (2020)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Learning Model Predictive Control for iterative tasks. A Data-Driven Control Framework](https://mlsanity.com/paper/1609.01387/) (2016)
- [Large Scale Model Predictive Control with Neural Networks and Primal Active Sets](https://mlsanity.com/paper/1910.10835/) (2019)
- [Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control](https://mlsanity.com/paper/2108.10315/) (2021)
- [Learning for MPC with Stability & Safety Guarantees](https://mlsanity.com/paper/2012.07369/) (2020)
- [Multiagent Rollout and Policy Iteration for POMDP with Application to Multi-Robot Repair Problems](https://mlsanity.com/paper/2011.04222/) (2020)
- [Learning Lyapunov Functions for Piecewise Affine Systems with Neural Network Controllers](https://mlsanity.com/paper/2008.06546/) (2020)
- [Multiagent Reinforcement Learning for Autonomous Routing and Pickup Problem with Adaptation to Variable Demand](https://mlsanity.com/paper/2211.14983/) (2022)
- [Neural Lyapunov Model Predictive Control: Learning Safe Global Controllers from Sub-optimal Examples](https://mlsanity.com/paper/2002.10451/) (2020)
- [Predictive Control with Learning-Based Terminal Costs Using Approximate Value Iteration](https://mlsanity.com/paper/2212.00361/) (2022)
- [A Regularized and Smoothed Fischer-Burmeister Method for Quadratic Programming with Applications to Model Predictive Control](https://mlsanity.com/paper/1807.03214/) (2018)
- [Data-driven Rollout for Deterministic Optimal Control](https://mlsanity.com/paper/2105.03116/) (2021)
- [Performance Bounds of Model Predictive Control for Unconstrained and Constrained Linear Quadratic Problems and Beyond](https://mlsanity.com/paper/2211.06187/) (2022)
- [A Forward Reachability Perspective on Control Barrier Functions and Discount Factors in Reachability Analysis](https://mlsanity.com/paper/2310.17180/) (2023)
- [Distributed Online Rollout for Multivehicle Routing in Unmapped Environments](https://mlsanity.com/paper/2305.15596/) (2023)
- [Most Likely Sequence Generation for n-Grams, Transformers, HMMs, and Markov Chains, by Using Rollout Algorithms](https://mlsanity.com/paper/2403.15465/) (2024)
- [Adaptive Horizon Model Predictive Control and Al'brekht's Method](https://mlsanity.com/paper/1904.00053/) (2019)
- [Operator Splitting for Adaptive Radiation Therapy with Nonlinear Health Dynamics](https://mlsanity.com/paper/2105.01286/) (2021)

## Built on by (most cited first, 7 of 7)

- [Improving Pre-Trained Vision-Language-Action Policies with Model-Based Search](https://mlsanity.com/paper/2508.12211/) (2025)
- [Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces](https://mlsanity.com/paper/2507.20853/) (2025)
- [Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regression](https://mlsanity.com/paper/2506.06954/) (2025)
- [Model-Free Predictive Control: Introductory Algebraic Calculations, and a Comparison with HEOL and ANNs](https://mlsanity.com/paper/2502.00443/) (2025)
- [Agentic Incident Response through Digital Twin-Enhanced Multiscale Planning](https://mlsanity.com/paper/2608.02422/) (2026)
- [On Architectures for Combining Reinforcement Learning and Model Predictive Control with Runtime Improvements](https://mlsanity.com/paper/2510.03354/) (2025)
- [Stable Transformer-Actor-Critic Model Predictive Control: A Contraction Analysis Approach](https://mlsanity.com/paper/2606.20197/) (2026)

## Abstract (arXiv)

In this paper we describe a new conceptual framework that connects approximate Dynamic Programming (DP), Model Predictive Control (MPC), and Reinforcement Learning (RL). This framework centers around two algorithms, which are designed largely independently of each other and operate in synergy through the powerful mechanism of Newton's method. We call them the off-line training and the on-line play algorithms. The names are borrowed from some of the major successes of RL involving games; primary examples are the recent (2017) AlphaZero program (which plays chess, [SHS17], [SSS17]), and the similarly structured and earlier (1990s) TD-Gammon program (which plays backgammon, [Tes94], [Tes95], [TeG96]). In these game contexts, the off-line training algorithm is the method used to teach the program how to evaluate positions and to generate good moves at any given position, while the on-line play algorithm is the method used to play in real time against human or computer opponents. Significantly, the synergy between off-line training and on-line play also underlies MPC (as well as other major classes of sequential decision problems), and indeed the MPC design architecture is very similar to the one of AlphaZero and TD-Gammon. This conceptual insight provides a vehicle for bridging the cultural gap between RL and MPC, and sheds new light on some fundamental issues in MPC. These include the enhancement of stability properties through rollout, the treatment of uncertainty through the use of certainty equivalence, the resilience of MPC in adaptive control settings that involve changing system parameters, and the insights provided by the superlinear performance bounds implied by Newton's method.