# Safe Reinforcement Learning with Chance-constrained Model Predictive Control

arXiv: https://arxiv.org/abs/2112.13941 · HTML: https://mlsanity.com/paper/2112.13941/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 27 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning](https://mlsanity.com/paper/1807.08364/) (2018)
- [Safe Reinforcement Learning Using Advantage-Based Intervention](https://mlsanity.com/paper/2106.09110/) (2021)
- [Linear model predictive safety certification for learning-based control](https://mlsanity.com/paper/1803.08552/) (2018)
- [Convex computation of the maximum controlled invariant set for polynomial control systems](https://mlsanity.com/paper/1303.6469/) (2013)

## Built on by (most cited first, 20 of 20)

- [A Survey of Constraint Formulations in Safe Reinforcement Learning](https://mlsanity.com/paper/2402.02025/) (2024)
- [Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms](https://mlsanity.com/paper/2310.03225/) (2023)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents](https://mlsanity.com/paper/2410.11671/) (2024)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Targeted Adversarial Attacks against Neural Network Trajectory Predictors](https://mlsanity.com/paper/2212.04138/) (2022)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [A Probabilistic Neuro-symbolic Layer for Algebraic Constraint Satisfaction](https://mlsanity.com/paper/2503.19466/) (2025)
- [Conformal Predictive Programming for Chance Constrained Optimization](https://mlsanity.com/paper/2402.07407/) (2024)
- [A Generalizable Physics-informed Learning Framework for Risk Probability Estimation](https://mlsanity.com/paper/2305.06432/) (2023)
- [Initial State Interventions for Deconfounded Imitation Learning](https://mlsanity.com/paper/2307.15980/) (2023)
- [A Retention-Centric Framework for Continual Learning with Guaranteed Model Developmental Safety](https://mlsanity.com/paper/2410.03955/) (2024)
- [Tempo Adaptation in Non-stationary Reinforcement Learning](https://mlsanity.com/paper/2309.14989/) (2023)
- [Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning](https://mlsanity.com/paper/2409.12045/) (2024)
- [Flipping-based Policy for Chance-Constrained Markov Decision Processes](https://mlsanity.com/paper/2410.06474/) (2024)
- [DeepSafeMPC: Deep Learning-Based Model Predictive Control for Safe Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.06397/) (2024)
- [Generalizable Physics-Informed Learning for Stochastic Safety-Critical Systems](https://mlsanity.com/paper/2407.08868/) (2024)
- [Finding Safe Zones of policies Markov Decision Processes](https://mlsanity.com/paper/2202.11593/) (2022)
- [Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping](https://mlsanity.com/paper/2607.23930/) (2026)
- [Learning Chance-Constrained MDPs with Bellman Distributional Certificates](https://mlsanity.com/paper/2609.30856/) (2026)

## Abstract (arXiv)

Real-world reinforcement learning (RL) problems often demand that agents behave safely by obeying a set of designed constraints. We address the challenge of safe RL by coupling a safety guide based on model predictive control (MPC) with a modified policy gradient framework in a linear setting with continuous actions. The guide enforces safe operation of the system by embedding safety requirements as chance constraints in the MPC formulation. The policy gradient training step then includes a safety penalty which trains the base policy to behave safely. We show theoretically that this penalty allows for a provably safe optimal base policy and illustrate our method with a simulated linearized quadrotor experiment.