mlsanityData to 14 Oct 2026

Iterative Reachability Estimation for Safe Reinforcement Learning

arXiv 2309.13528 ↗First posted 24 Sep 2023cs.LG cs.AI cs.ROcited by 37

Not on the map. Why?

Cited by 28 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Ensuring safety is important for the practical deployment of reinforcement learning (RL). Various challenges must be addressed, such as handling stochasticity in the environments, providing rigorous guarantees of persistent state-wise safety satisfaction, and avoiding overly conservative behaviors that sacrifice performance. We propose a new framework, Reachability Estimation for Safe Policy Optimization (RESPO), for safety-constrained RL in general stochastic settings. In the feasible set where there exist violation-free policies, we optimize for rewards while maintaining persistent safety. Outside this feasible set, our optimization produces the safest behavior by guaranteeing entrance into the feasible set whenever possible with the least cumulative discounted violations. We introduce a class of algorithms using our novel reachability estimation function to optimize in our proposed framework and in similar frameworks such as those concurrently handling multiple hard and soft constraints. We theoretically establish that our algorithms almost surely converge to locally optimal policies of our safe optimization framework. We evaluate the proposed methods on a diverse suite of safe RL environments from Safety Gym, PyBullet, and MuJoCo, and show the benefits in improving both reward performance and safety compared with state-of-the-art baselines.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

42 of 58 references.

References · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2018Reward Constrained Policy Optimizationcited by 417
  5. 2019Control Barrier Functions: Theory and Applicationscited by 372
  6. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  7. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  8. 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 287
  9. 2020Projection-Based Constrained Policy Optimizationcited by 203
  10. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 201
  11. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  12. 2022Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methodscited by 151
30 more
  1. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  2. 2020Neural Lyapunov Controlcited by 142
  3. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  4. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 91
  5. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  6. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  7. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  8. 2022Reachability Constrained Reinforcement Learningcited by 59
  9. 2021Robust Control Barrier-Value Functions for Safety-Critical Controlcited by 51
  10. 2014Reach-Avoid Problems with Time-Varying Dynamics, Targets and Constraintscited by 44
  11. 2021Stabilizing Neural Control Using Self-Learned Almost Lyapunov Criticscited by 35
  12. 2016Robustness of Control Barrier Functions for Safety Critical Controlcited by 30
  13. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  14. 2022Safe Reinforcement Learning Using Black-Box Reachability Analysiscited by 26
  15. 2022Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopescited by 26
  16. 2021Density Constrained Reinforcement Learningcited by 25
  17. 2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 20
  18. 2016Decomposition of Reachable Sets and Tubes for a Class of Nonlinear Systemscited by 18
  19. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 18
  20. 2021Risk-sensitive safety analysis using Conditional Value-at-Riskcited by 14
  21. 2022Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systemscited by 14
  22. 2019Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraintscited by 12
  23. 2021A Simple and Efficient Sampling-based Algorithm for General Reachability Analysiscited by 12
  24. 2018A Minimum Discounted Reward Hamilton-Jacobi Formulation for Computing Reachable Setscited by 11
  25. 2023Provably Safe Reinforcement Learning with Step-wise Violation Constraintscited by 11
  26. 2022Quantifying Safety of Learning-based Self-Driving Control Using Almost-Barrier Functionscited by 10
  27. 2023Learning Stabilization Control from Observations by Learning Lyapunov-like Proxy Modelscited by 10
  28. 2019Reachability-Based Safety Guarantees using Efficient Initializationscited by 9
  29. 2021Model-Free Safety-Critical Control for Robotic Systemscited by 8
  30. 2018Stochastic reachability of a target tube: Theory and computationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 19.

37 papers.

Sort:

By area:

  1. 2025Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failurescited by 33
  2. 2024Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificatescited by 16
  3. 2024Hamilton-Jacobi Reachability in Reinforcement Learning: A Surveycited by 16
  4. 2024Solving Minimum-Cost Reach Avoid using Reinforcement Learningcited by 14
  5. 2025Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoningcited by 10
  6. 2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulationcited by 9
  7. 2025Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Controlcited by 8
  8. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  9. 2024Verified Safe Reinforcement Learning for Neural Network Dynamic Modelscited by 7
  10. 2025Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARLcited by 6
  11. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 5
  12. 2024Safe and Reliable Training of Learning-Based Aerospace Controllerscited by 4
25 more
  1. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  2. 2025Provably Optimal Reinforcement Learning under Safety Filteringcited by 3
  3. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  4. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
  5. 2024Extremum-Seeking Action Selection for Accelerating Policy Optimizationcited by 1
  6. 2025Learning Vision-Based Neural Network Controllers with Semi-Probabilistic Safety Guaranteescited by 1
  7. 2026Certifying Hamilton-Jacobi Reachability Learned via Reinforcement Learningcited by 1
  8. 2026Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertaintycited by 1
  9. 2026Augmented Lagrangian Multiplier Network for State-wise Safety in Reinforcement Learningcited by 1
  10. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 1
  11. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  12. 2025Constraint-Aware Reinforcement Learning via Adaptive Action Scalingcited by 0
  13. 2026Unifying Hamilton-Jacobi Reachability and Reinforcement Learningcited by 0
  14. 2026Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learningcited by 0
  15. 2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Managementcited by 0
  16. 2026COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenancecited by 0
  17. 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
  18. 2026Stochastic Minimum-Cost Reach-Avoid Reinforcement Learningcited by 0
  19. 2026Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Controlcited by 0
  20. 2026Robust Peak-cost Constrained Reinforcement Learningcited by 0
  21. 2026Constrained Flow Policy Updates: A Generalized Schr"odinger Bridge Viewcited by 0
  22. 2026Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learningcited by 0
  23. 2026Reachability-Informed Reinforcement Learning for Multi-Impulse Interplanetary Transferscited by 0
  24. 2026Instance-Dependent Regret for CMDPs with Step-Wise Constraintscited by 0
  25. 2026Reachability-Aware Diffusion Policy Optimizationcited by 0