# MPC4RL -- A Software Package for Reinforcement Learning based on Model Predictive Control

arXiv: https://arxiv.org/abs/2501.15897 · HTML: https://mlsanity.com/paper/2501.15897/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 11 of 11)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [acados: a modular open-source framework for fast embedded optimal control](https://mlsanity.com/paper/1910.13753/) (2019)
- [HPIPM: a high-performance quadratic programming framework for model predictive control](https://mlsanity.com/paper/2003.02547/) (2020)
- [MPC-based Reinforcement Learning for Economic Problems with Application to Battery Storage](https://mlsanity.com/paper/2104.02411/) (2021)
- [MPC-based Reinforcement Learning for a Simplified Freight Mission of Autonomous Surface Vehicles](https://mlsanity.com/paper/2106.08634/) (2021)
- [Equivalence of Optimality Criteria for Markov Decision Process and Model Predictive Control](https://mlsanity.com/paper/2210.04302/) (2022)

## Built on by (most cited first, 2 of 2)

- [Differentiable Nonlinear Model Predictive Control](https://mlsanity.com/paper/2505.01353/) (2025)
- [Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents](https://mlsanity.com/paper/2507.13491/) (2025)

## Abstract (arXiv)

In this paper, we present an early software integrating Reinforcement Learning (RL) with Model Predictive Control (MPC). Our aim is to make recent theoretical contributions from the literature more accessible to both the RL and MPC communities. We combine standard software tools developed by the RL community, such as Gymnasium, stable-baselines3, or CleanRL with the acados toolbox, a widely-used software package for efficient MPC algorithms. Our core contribution is MPC4RL, an open-source Python package that supports learning-enhanced MPC schemes for existing acados implementations. The package is designed to be modular, extensible, and user-friendly, facilitating the tuning of MPC algorithms for a broad range of control problems. It is available on GitHub.