mlsanityData to 14 Oct 2026

Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning

arXiv 2405.02754 ↗First posted 4 May 2024cs.RO cs.AI cs.LGcited by 3

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Deep reinforcement learning (DRL) has demonstrated remarkable performance in many continuous control tasks. However, a significant obstacle to the real-world application of DRL is the lack of safety guarantees. Although DRL agents can satisfy system safety in expectation through reward shaping, designing agents to consistently meet hard constraints (e.g., safety specifications) at every time step remains a formidable challenge. In contrast, existing work in the field of safe control provides guarantees on persistent satisfaction of hard safety constraints. However, these methods require explicit analytical system dynamics models to synthesize safe control, which are typically inaccessible in DRL settings. In this paper, we present a model-free safe control algorithm, the implicit safe set algorithm, for synthesizing safeguards for DRL agents that ensure provable safety throughout training. The proposed algorithm synthesizes a safety index (barrier certificate) and a subsequent safe control law solely by querying a black-box dynamic function (e.g., a digital twin simulator). Moreover, we theoretically prove that the implicit safe set algorithm guarantees finite time convergence to the safe set and forward invariance for both continuous-time and discrete-time systems. We validate the proposed algorithm on the state-of-the-art Safety Gym benchmark, where it achieves zero safety violations while gaining 95% ± 9% cumulative reward compared to state-of-the-art safe DRL methods. Furthermore, the resulting algorithm scales well to high-dimensional systems with parallel computing.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 78 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

32 of 78 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  4. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  5. 2018Safe Exploration in Continuous Action Spacescited by 247
  6. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  7. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  8. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 70
  9. 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 59
  10. 2023State-wise Safe Reinforcement Learning: A Surveycited by 55
  11. 2019Adaptive Safety with Control Barrier Functionscited by 25
  12. 2019Smoothing Policies and Safe Policy Gradientscited by 21
20 more
  1. 2020ShieldNN: A Provably Safe NN Filter for Unsafe NN Controllerscited by 17
  2. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 17
  3. 2022Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Modelscited by 15
  4. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  5. 2023Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functionscited by 13
  6. 2023GUARD: A Safe Reinforcement Learning Benchmarkcited by 13
  7. 2022Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programmingcited by 12
  8. 2022Safety Index Synthesis via Sum-of-Squares Programmingcited by 12
  9. 2023State-wise Constrained Policy Optimizationcited by 9
  10. 2019Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithmcited by 8
  11. 2019Stochastic Variance Reduction for Deep Q-learningcited by 7
  12. 2020Experimental Evaluation of Human Motion Prediction: Toward Safe and Efficient Human Robot Collaborationcited by 7
  13. 2020Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimizationcited by 7
  14. 2021Provably Safe Tolerance Estimation for Robot Arms via Sum-of-Squares Programmingcited by 6
  15. 2023Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimizationcited by 4
  16. 2024Real-time Safety Index Adaptation for Parameter-varying Systems via Determinant Gradient Ascendcited by 4
  17. 2023Hybrid Task Constrained Planner for Robot Manipulator in Confined Environmentcited by 3
  18. 2023Safety Index Synthesis with State-dependent Control Spacecited by 3
  19. 2023Learning Predictive Safety Filter via Decomposition of Robust Invariant Setcited by 3
  20. 2022Edge-assisted Collaborative Digital Twin for Safety-Critical Robotics in Industrial IoTcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2026Provably Safe, Yet Scalable Reinforcement Learningcited by 1
  2. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfactioncited by 0
  3. 2026Can Explicit Physical Feasibility Benefit VLA Learning? An Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe Reinforcement Learning using Ideas from Model Predictive Control
  2. 2024Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems
  3. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applications
  4. 2020Safety Considerations in Deep Control Policies with Safety Barrier Certificates Under Uncertainty
  5. 2026SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning
  6. 2026Sampling-Based Safe Reinforcement Learning
  7. 2024Verification-Guided Shielding for Deep Reinforcement Learning
  8. 2024Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning