# Safe Reinforcement Learning Using Black-Box Reachability Analysis

arXiv: https://arxiv.org/abs/2204.07417 · HTML: https://mlsanity.com/paper/2204.07417/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 30 of 33)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Exploration-Exploitation in Constrained MDPs](https://mlsanity.com/paper/2003.02189/) (2020)
- [Socially Aware Motion Planning with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.08862/) (2017)
- [Provably Efficient Safe Exploration via Primal-Dual Policy Optimization](https://mlsanity.com/paper/2003.00534/) (2020)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.07845/) (2016)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation](https://mlsanity.com/paper/1703.00420/) (2017)
- [Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/1709.10082/) (2017)
- [Socially Compliant Navigation through Raw Depth Inputs with Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1710.02543/) (2017)
- [Safe Reinforcement Learning Using Advantage-Based Intervention](https://mlsanity.com/paper/2106.09110/) (2021)
- [Value constrained model-free continuous control](https://mlsanity.com/paper/1902.04623/) (2019)
- [Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Control](https://mlsanity.com/paper/2011.08421/) (2020)
- [On Infusing Reachability-Based Safety Assurance within Planning Frameworks for Human-Robot Vehicle Interactions](https://mlsanity.com/paper/2012.03390/) (2020)
- [Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Framework](https://mlsanity.com/paper/2008.11700/) (2020)
- [Bridging the Gap Between Safety and Real-Time Performance in Receding-Horizon Trajectory Design for Mobile Robots](https://mlsanity.com/paper/1809.06746/) (2018)
- [Data-Driven Reachability Analysis Using Matrix Zonotopes](https://mlsanity.com/paper/2011.08472/) (2020)
- [Model-Free Safety-Critical Control for Robotic Systems](https://mlsanity.com/paper/2109.09047/) (2021)
- [Constrained Feedforward Neural Network Training via Reachability Analysis](https://mlsanity.com/paper/2107.07696/) (2021)
- [Control Barrier Functions With Unmodeled Dynamics Using Integral Quadratic Constraints](https://mlsanity.com/paper/2108.10491/) (2021)

## Built on by (most cited first, 26 of 26)

- [Iterative Reachability Estimation for Safe Reinforcement Learning](https://mlsanity.com/paper/2309.13528/) (2023)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.13863/) (2024)
- [SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learning](https://mlsanity.com/paper/2407.17460/) (2024)
- [Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysis](https://mlsanity.com/paper/2503.03911/) (2025)
- [Safe Reinforcement Learning using Data-Driven Predictive Control](https://mlsanity.com/paper/2211.11027/) (2022)
- [RAIL: Reachability-Aided Imitation Learning for Safe Policy Execution](https://mlsanity.com/paper/2409.19190/) (2024)
- [Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learning](https://mlsanity.com/paper/2408.14063/) (2024)
- [Do No Harm: A Counterfactual Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2405.11669/) (2024)
- [AI-Powered CPS-Enabled Vulnerable-User-Aware Urban Transportation Digital Twin: Methods and Applications](https://mlsanity.com/paper/2501.10396/) (2024)
- [Safe Learning Control with Optimality and Stability Guarantees](https://mlsanity.com/paper/2501.15373/) (2025)
- [Leveraging Analytic Gradients in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2506.01665/) (2025)
- [Predictive Safety Shield for Dyna-Q Reinforcement Learning](https://mlsanity.com/paper/2511.21531/) (2025)
- [Safe Deep RL for Intraoperative Planning of Pedicle Screw Placement](https://mlsanity.com/paper/2305.05354/) (2023)
- [Adaptive Policy Learning to Additional Tasks](https://mlsanity.com/paper/2305.15193/) (2023)
- [Enforcing Cooperative Safety for Reinforcement Learning-based Mixed-Autonomy Platoon Control](https://mlsanity.com/paper/2411.10031/) (2024)
- [Safely Learning Controlled Stochastic Dynamics](https://mlsanity.com/paper/2506.02754/) (2025)
- [A Control-Barrier-Function-Based Algorithm for Policy Adaptation in Reinforcement Learning](https://mlsanity.com/paper/2510.02720/) (2025)
- [Constraint-Aware Reinforcement Learning via Adaptive Action Scaling](https://mlsanity.com/paper/2510.11491/) (2025)
- [Reversible Simplex Supervision with Post-Action Debt Accounting for Goal-Reaching RL](https://mlsanity.com/paper/2601.19499/) (2026)
- [Conformal Reachability for Safe Control in Unknown Environments](https://mlsanity.com/paper/2602.03799/) (2026)
- [Safe Continual Reinforcement Learning in Non-stationary Environments](https://mlsanity.com/paper/2604.19737/) (2026)
- [Towards Safe Reinforcement Learning with Reduced Conservativeness: A Case Study on Drone Flight Control](https://mlsanity.com/paper/2608.26852/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is capable of sophisticated motion planning and control for robots in uncertain environments. However, state-of-the-art deep RL approaches typically lack safety guarantees, especially when the robot and environment models are unknown. To justify widespread deployment, robots must respect safety constraints without sacrificing performance. Thus, we propose a Black-box Reachability-based Safety Layer (BRSL) with three main components: (1) data-driven reachability analysis for a black-box robot model, (2) a trajectory rollout planner that predicts future actions and observations using an ensemble of neural networks trained online, and (3) a differentiable polytope collision check between the reachable set and obstacles that enables correcting unsafe actions. In simulation, BRSL outperforms other state-of-the-art safe RL methods on a Turtlebot 3, a quadrotor, a trajectory-tracking point mass, and a hexarotor in wind with an unsafe set adjacent to the area of highest reward.