# Safe Reinforcement Learning using Data-Driven Predictive Control

arXiv: https://arxiv.org/abs/2211.11027 · HTML: https://mlsanity.com/paper/2211.11027/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 20 Nov 2022.

## Builds on (most cited first, 24 of 24)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Safe Exploration in Finite Markov Decision Processes with Gaussian Processes](https://mlsanity.com/paper/1606.04753/) (2016)
- [On the role of planning in model-based deep reinforcement learning](https://mlsanity.com/paper/2011.04021/) (2020)
- [Combating the Compounding-Error Problem with a Multi-step Model](https://mlsanity.com/paper/1905.13320/) (2019)
- [Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning](https://mlsanity.com/paper/2205.11790/) (2022)
- [Safe Reinforcement Learning Using Advantage-Based Intervention](https://mlsanity.com/paper/2106.09110/) (2021)
- [Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Control](https://mlsanity.com/paper/2011.08421/) (2020)
- [Safe Reinforcement Learning Using Black-Box Reachability Analysis](https://mlsanity.com/paper/2204.07417/) (2022)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions](https://mlsanity.com/paper/2012.03390/) (2020)
- [Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Framework](https://mlsanity.com/paper/2008.11700/) (2020)
- [Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret](https://mlsanity.com/paper/1505.05798/) (2015)
- [Data-Driven Reachability Analysis from Noisy Data](https://mlsanity.com/paper/2105.07229/) (2021)
- [Bridging the Gap Between Safety and Real-Time Performance in Receding-Horizon Trajectory Design for Mobile Robots](https://mlsanity.com/paper/1809.06746/) (2018)
- [Data-Driven Reachability Analysis Using Matrix Zonotopes](https://mlsanity.com/paper/2011.08472/) (2020)
- [Model-Free Safety-Critical Control for Robotic Systems](https://mlsanity.com/paper/2109.09047/) (2021)
- [Control Barrier Functions With Unmodeled Dynamics Using Integral Quadratic Constraints](https://mlsanity.com/paper/2108.10491/) (2021)
- [Safely Probabilistically Complete Real-Time Planning and Exploration in Unknown Environments](https://mlsanity.com/paper/1811.07834/) (2018)
- [Robust Data-Driven Predictive Control using Reachability Analysis](https://mlsanity.com/paper/2103.14110/) (2021)

## Built on by (most cited first, 5 of 5)

- [Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysis](https://mlsanity.com/paper/2503.03911/) (2025)
- [Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?](https://mlsanity.com/paper/2509.12833/) (2025)
- [X-Safe: Embodiment-Agnostic Safeguarding for Robot Arm Manipulation Policies via Action Masking](https://mlsanity.com/paper/2606.22278/) (2026)
- [Leveraging Analytic Gradients in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2506.01665/) (2025)
- [Safe Reinforcement Learning with a Predictive Safety Filter for Motion Planning and Control: A Drifting Vehicle Example](https://mlsanity.com/paper/2506.22894/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms can achieve state-of-the-art performance in decision-making and continuous control tasks. However, applying RL algorithms on safety-critical systems still needs to be well justified due to the exploration nature of many RL algorithms, especially when the model of the robot and the environment are unknown. To address this challenge, we propose a data-driven safety layer that acts as a filter for unsafe actions. The safety layer uses a data-driven predictive controller to enforce safety guarantees for RL policies during training and after deployment. The RL agent proposes an action that is verified by computing the data-driven reachability analysis. If there is an intersection between the reachable set of the robot using the proposed action, we call the data-driven predictive controller to find the closest safe action to the proposed unsafe action. The safety layer penalizes the RL agent if the proposed action is unsafe and replaces it with the closest safe one. In the simulation, we show that our method outperforms state-of-the-art safe RL methods on the robotics navigation problem for a Turtlebot 3 in Gazebo and a quadrotor in Unreal Engine 4 (UE4).