# Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification

arXiv: https://arxiv.org/abs/2502.02133 · HTML: https://mlsanity.com/paper/2502.02133/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 138 of its 331 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 30 of 138)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon](https://mlsanity.com/paper/1811.06128/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023)
- [Cautious Model Predictive Control using Gaussian Process Regression](https://mlsanity.com/paper/1705.10702/) (2017)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)

## Built on by (most cited first, 26 of 26)

- [Differentiable Nonlinear Model Predictive Control](https://mlsanity.com/paper/2505.01353/) (2025)
- [Probabilistically safe and efficient model-based reinforcement learning](https://mlsanity.com/paper/2504.00626/) (2025)
- [MPCritic: A plug-and-play MPC architecture for reinforcement learning](https://mlsanity.com/paper/2504.01086/) (2025)
- [Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation](https://mlsanity.com/paper/2606.05687/) (2026)
- [Local-Global Learning of Interpretable Control Policies: The Interface between MPC and Reinforcement Learning](https://mlsanity.com/paper/2503.13289/) (2025)
- [Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimality](https://mlsanity.com/paper/2510.17709/) (2025)
- [Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisions](https://mlsanity.com/paper/2511.15830/) (2025)
- [MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins](https://mlsanity.com/paper/2606.26392/) (2026)
- [Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks](https://mlsanity.com/paper/2608.20858/) (2026)
- [Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents](https://mlsanity.com/paper/2507.13491/) (2025)
- [Computationally efficient Gauss-Newton reinforcement learning for model predictive control](https://mlsanity.com/paper/2508.02441/) (2025)
- [Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control](https://mlsanity.com/paper/2509.15799/) (2025)
- [Direct transfer of optimized controllers to similar systems using dimensionless MPC](https://mlsanity.com/paper/2512.08667/) (2025)
- [CORL: Reinforcement Learning of MILP Policies Solved via Branch and Bound](https://mlsanity.com/paper/2512.11169/) (2025)
- [Safety-Guaranteed Imitation Learning from Nonlinear Model Predictive Control for Spacecraft Close Proximity Operations](https://mlsanity.com/paper/2603.18910/) (2026)
- [Bridging RL and MPC for mixed-integer optimal control with application to Formula 1 race strategies](https://mlsanity.com/paper/2604.00826/) (2026)
- [Soft MPCritic: Value-Based Reinforcement Learning for Non-Differentiable Predictive Control](https://mlsanity.com/paper/2604.01477/) (2026)
- [A Systematic Review and Taxonomy of Reinforcement Learning-Model Predictive Control Integration for Linear Systems](https://mlsanity.com/paper/2604.21030/) (2026)
- [Application of Deep Reinforcement Learning to Event-Triggered Control for Networked Artificial Pancreas Systems](https://mlsanity.com/paper/2604.26126/) (2026)
- [Solving Markov Decision Processes with Future Information via MPC](https://mlsanity.com/paper/2606.24991/) (2026)
- [Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping](https://mlsanity.com/paper/2607.23930/) (2026)
- [SelectLight: Learning to Select Signal Plans Generated by Distributed Model Predictive Control for Urban Traffic Networks](https://mlsanity.com/paper/2608.12256/) (2026)
- [Guided Riemannian Optimization (GuRO): Bridging Model Predictive Control and Decision Transformers](https://mlsanity.com/paper/2608.23204/) (2026)
- [Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC](https://mlsanity.com/paper/2609.01061/) (2026)
- [Driving Context-guided Model Predictive Planning and Control for Autonomous Car Racing at the Limit and Beyond](https://mlsanity.com/paper/2609.12660/) (2026)
- [Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control](https://mlsanity.com/paper/2609.17697/) (2026)

## Abstract (arXiv)

The fields of MPC and RL consider two successful control techniques for Markov decision processes. Both approaches are derived from similar fundamental principles, and both are widely used in practical applications, including robotics, process control, energy systems, and autonomous driving. Despite their similarities, MPC and RL follow distinct paradigms that emerged from diverse communities and different requirements. Various technical discrepancies, particularly the role of an environment model as part of the algorithm, lead to methodologies with nearly complementary advantages. Due to their orthogonal benefits, research interest in combination methods has recently increased significantly, leading to a large and growing set of complex ideas leveraging MPC and RL. This work illuminates the differences, similarities, and fundamentals that allow for different combination algorithms and categorizes existing work accordingly. Particularly, we focus on the versatile actor-critic RL approach as a basis for our categorization and examine how the online optimization approach of MPC can be used to improve the overall closed-loop performance of a policy.