# Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.02754 · HTML: https://mlsanity.com/paper/2405.02754/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 32 of its 78 references that are in the map.
- First posted on arXiv 4 May 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [State-wise Safe Reinforcement Learning: A Survey](https://mlsanity.com/paper/2302.03122/) (2023)
- [Adaptive Safety with Control Barrier Functions](https://mlsanity.com/paper/1910.00555/) (2019)
- [Smoothing Policies and Safe Policy Gradients](https://mlsanity.com/paper/1905.03231/) (2019)
- [ShieldNN: A Provably Safe NN Filter for Unsafe NN Controllers](https://mlsanity.com/paper/2006.09564/) (2020)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)
- [Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Models](https://mlsanity.com/paper/2210.01041/) (2022)
- [Risk-Sensitive Reinforcement Learning with Exponential Criteria](https://mlsanity.com/paper/2212.09010/) (2022)
- [Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions](https://mlsanity.com/paper/2305.09793/) (2023)
- [GUARD: A Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2305.13681/) (2023)
- [Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programming](https://mlsanity.com/paper/2209.06896/) (2022)
- [Safety Index Synthesis via Sum-of-Squares Programming](https://mlsanity.com/paper/2209.09134/) (2022)
- [State-wise Constrained Policy Optimization](https://mlsanity.com/paper/2306.12594/) (2023)
- [Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithm](https://mlsanity.com/paper/1912.09095/) (2019)
- [Stochastic Variance Reduction for Deep Q-learning](https://mlsanity.com/paper/1905.08152/) (2019)
- [Experimental Evaluation of Human Motion Prediction: Toward Safe and Efficient Human Robot Collaboration](https://mlsanity.com/paper/2001.09550/) (2020)
- [Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimization](https://mlsanity.com/paper/2008.03826/) (2020)
- [Provably Safe Tolerance Estimation for Robot Arms via Sum-of-Squares Programming](https://mlsanity.com/paper/2104.08896/) (2021)
- [Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimization](https://mlsanity.com/paper/2308.13140/) (2023)
- [Real-time Safety Index Adaptation for Parameter-varying Systems via Determinant Gradient Ascend](https://mlsanity.com/paper/2403.14968/) (2024)
- [Hybrid Task Constrained Planner for Robot Manipulator in Confined Environment](https://mlsanity.com/paper/2304.09260/) (2023)
- [Safety Index Synthesis with State-dependent Control Space](https://mlsanity.com/paper/2309.12406/) (2023)

## Built on by (most cited first, 3 of 3)

- [Provably Safe, Yet Scalable Reinforcement Learning](https://mlsanity.com/paper/2606.14536/) (2026)
- [Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction](https://mlsanity.com/paper/2410.01212/) (2024)
- [Can Explicit Physical Feasibility Benefit VLA Learning? An Empirical Study](https://mlsanity.com/paper/2604.17896/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has demonstrated remarkable performance in many continuous control tasks. However, a significant obstacle to the real-world application of DRL is the lack of safety guarantees. Although DRL agents can satisfy system safety in expectation through reward shaping, designing agents to consistently meet hard constraints (e.g., safety specifications) at every time step remains a formidable challenge. In contrast, existing work in the field of safe control provides guarantees on persistent satisfaction of hard safety constraints. However, these methods require explicit analytical system dynamics models to synthesize safe control, which are typically inaccessible in DRL settings. In this paper, we present a model-free safe control algorithm, the implicit safe set algorithm, for synthesizing safeguards for DRL agents that ensure provable safety throughout training. The proposed algorithm synthesizes a safety index (barrier certificate) and a subsequent safe control law solely by querying a black-box dynamic function (e.g., a digital twin simulator). Moreover, we theoretically prove that the implicit safe set algorithm guarantees finite time convergence to the safe set and forward invariance for both continuous-time and discrete-time systems. We validate the proposed algorithm on the state-of-the-art Safety Gym benchmark, where it achieves zero safety violations while gaining 95% ± 9% cumulative reward compared to state-of-the-art safe DRL methods. Furthermore, the resulting algorithm scales well to high-dimensional systems with parallel computing.