mlsanityData to 7 Oct 2026

Reachability Constrained Reinforcement Learning

arXiv 2205.07536 ↗First posted 16 May 2022cs.LG cs.AI cs.ROcited by 52

Not on the map. Why?

Abstract (arXiv)

Constrained reinforcement learning (CRL) has gained significant interest recently, since safety constraints satisfaction is critical for real-world problems. However, existing CRL methods constraining discounted cumulative costs generally lack rigorous definition and guarantee of safety. In contrast, in the safe control research, safety is defined as persistently satisfying certain state constraints. Such persistent safety is possible only on a subset of the state space, called feasible set, where an optimal largest feasible set exists for a given environment. Recent studies incorporate feasible sets into CRL with energy-based methods such as control barrier function (CBF), safety index (SI), and leverage prior conservative estimations of feasible sets, which harms the performance of the learned policy. To deal with this problem, this paper proposes the reachability CRL (RCRL) method by using reachability analysis to establish the novel self-consistency condition and characterize the feasible sets. The feasible sets are represented by the safety value function, which is used as the constraint in CRL. We use the multi-time scale stochastic approximation theory to prove that the proposed algorithm converges to a local optimum, where the largest feasible set can be guaranteed. Empirical results on different benchmarks validate the learned feasible set, the policy performance, and constraint satisfaction of RCRL, compared to CRL and safe control baselines.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

32 of 49 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2018Reward Constrained Policy Optimizationcited by 397
  7. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  9. 2019Control Barrier Functions: Theory and Applicationscited by 332
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  11. 2020Projection-Based Constrained Policy Optimizationcited by 195
  12. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
20 more
  1. 2020The Lipschitz Constant of Self-Attentioncited by 171
  2. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  3. 2020Conservative Safety Critics for Explorationcited by 107
  4. 2021Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functionscited by 91
  5. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 81
  6. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  7. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  8. 2021Robust Control Barrier-Value Functions for Safety-Critical Controlcited by 47
  9. 2021Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learningcited by 43
  10. 2021safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Roboticscited by 42
  11. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  12. 2021Model-based Constrained Reinforcement Learning using Generalized Control Barrier Functioncited by 30
  13. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  14. 2021Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligencecited by 25
  15. 2021Density Constrained Reinforcement Learningcited by 25
  16. 2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 25
  17. 2021Safe Autonomous Racing via Approximate Reachability on Ego-visioncited by 13
  18. 2019Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraintscited by 12
  19. 2021Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and modelcited by 8
  20. 2019Duality between density function and value function with applications in constrained optimal control and Markov Decision Processcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 8; 2024: 17; 2025: 9; 2026: 13.

52 papers.

Sort:

By area:

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  2. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  3. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  4. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18
  5. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 16
  6. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  7. 2024Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificatescited by 15
  8. 2024Solving Minimum-Cost Reach Avoid using Reinforcement Learningcited by 14
  9. 2024FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversarialitycited by 13
  10. 2024Hamilton-Jacobi Reachability in Reinforcement Learning: A Surveycited by 11
  11. 2023Provably Safe Reinforcement Learning with Step-wise Violation Constraintscited by 10
  12. 2023Safe Reinforcement Learning with Dual Robustnesscited by 10
38 more
  1. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  2. 2024To the Max: Reinventing Reward in Reinforcement Learningcited by 7
  3. 2024Verified Safe Reinforcement Learning for Neural Network Dynamic Modelscited by 7
  4. 2024Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systemscited by 6
  5. 2024Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibriumcited by 5
  6. 2025SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulationcited by 5
  7. 2023Feasible Policy Iteration for Safe Reinforcement Learningcited by 4
  8. 2023Robust Safe Reinforcement Learning under Adversarial Disturbancescited by 4
  9. 2024Safe and Reliable Training of Learning-Based Aerospace Controllerscited by 4
  10. 2025Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARLcited by 4
  11. 2025Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systemscited by 4
  12. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 4
  13. 2022Safe Reinforcement Learning with Contrastive Risk Predictioncited by 3
  14. 2022Guiding Safe Exploration with Weakest Preconditionscited by 3
  15. 2023Learning Predictive Safety Filter via Decomposition of Robust Invariant Setcited by 3
  16. 2024Stable and Safe Human-aligned Reinforcement Learning through Neural Ordinary Differential Equationscited by 3
  17. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
  18. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  19. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  20. 2023MPC-Inspired Reinforcement Learning for Verifiable Model-Free Controlcited by 2
  21. 2025Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulationscited by 2
  22. 2024Policy Bifurcation in Safe Reinforcement Learningcited by 1
  23. 2025Learning Vision-Based Neural Network Controllers with Semi-Probabilistic Safety Guaranteescited by 1
  24. 2025Accelerated Learning with Linear Temporal Logic using Differentiable Simulationcited by 1
  25. 2026Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 1
  26. 2026Provably Safe, Yet Scalable Reinforcement Learningcited by 1
  27. 2024Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learningcited by 0
  28. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfactioncited by 0
  29. 2025Game-Theoretic Risk-Shaped Reinforcement Learning for Safe Autonomous Drivingcited by 0
  30. 2025Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 0
  31. 2026Progress Constraints for Reinforcement Learning in Behavior Treescited by 0
  32. 2026Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertaintycited by 0
  33. 2026Safe Reinforcement Learning with Preference-based Constraint Inferencecited by 0
  34. 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
  35. 2026Stochastic Minimum-Cost Reach-Avoid Reinforcement Learningcited by 0
  36. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  37. 2026Robust Peak-cost Constrained Reinforcement Learningcited by 0
  38. 2026Self-Aware Active Learning Enables Continual Improvement in Autonomous Drivingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
  2. 2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning
  3. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
  4. 2021Safe Continuous Control with Constrained Model-Based Policy Optimization
  5. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  6. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning
  7. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
  8. 2026Safe Meta-Reinforcement Learning via Information Space Reachability