mlsanityData to 14 Oct 2026

Probabilistic Constraint for Safety-Critical Reinforcement Learning

arXiv 2306.17279 ↗First posted 29 Jun 2023cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider the problem of learning safe policies for probabilistic-constrained reinforcement learning (RL). Specifically, a safe policy or controller is one that, with high probability, maintains the trajectory of the agent in a given safe set. We establish a connection between this probabilistic-constrained setting and the cumulative-constrained formulation that is frequently explored in the existing literature. We provide theoretical bounds elucidating that the probabilistic-constrained setting offers a better trade-off in terms of optimality and safety (constraint satisfaction). The challenge encountered when dealing with the probabilistic constraints, as explored in this work, arises from the absence of explicit expressions for their gradients. Our prior work provides such an explicit gradient expression for probabilistic constraints which we term Safe Policy Gradient-REINFORCE (SPG-REINFORCE). In this work, we provide an improved gradient SPG-Actor-Critic that leads to a lower variance than SPG-REINFORCE, which is substantiated by our theoretical results. A noteworthy aspect of both SPGs is their inherent algorithm independence, rendering them versatile for application across a range of policy-based algorithms. Furthermore, we propose a Safe Primal-Dual algorithm that can leverage both SPGs to learn safe policies. It is subsequently followed by theoretical analyses that encompass the convergence of the algorithm, as well as the near-optimality and feasibility on average. In addition, we test the proposed approaches by a series of empirical experiments. These experiments aim to examine and analyze the inherent trade-offs between the optimality and safety, and serve to substantiate the efficacy of two SPGs, as well as our theoretical contributions.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 29 of its 72 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

29 of 72 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2021Convex optimizationcited by 2,515
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  8. 2016OpenAI Gymcited by 1,736
  9. 2017Constrained Policy Optimizationcited by 983
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  11. 2018Reward Constrained Policy Optimizationcited by 417
  12. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
17 more
  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  2. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  3. 2020Projection-Based Constrained Policy Optimizationcited by 203
  4. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  5. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  6. 2020Conservative Safety Critics for Explorationcited by 113
  7. 2017AI Safety Gridworldscited by 110
  8. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  9. 2017Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigationcited by 77
  10. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  11. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  12. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  13. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 47
  14. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  15. 2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 40
  16. 2022Policy Gradients for Probabilistic Constrained Reinforcement Learningcited by 5
  17. 2021Towards Safe Continuing Task Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 0; 2026: 3.

11 papers.

  1. 2023Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Controlcited by 20
  2. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 15
  3. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
  4. 2024Adaptive Primal-Dual Method for Safe Reinforcement Learningcited by 4
  5. 2024Learning safety critics via a non-contractive binary bellman operatorcited by 3
  6. 2023Absolute Policy Optimizationcited by 2
  7. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithmscited by 1
  8. 2024Transfer Learning for a Class of Cascade Dynamical Systemscited by 1
  9. 2026Safety Guarantees in Zero-Shot Reinforcement Learning for Cascade Dynamical Systemscited by 0
  10. 2026Learning-Based Stochastic Optimal Control with Infinite-Horizon Probabilistic Constraintscited by 0
  11. 2026Learning Chance-Constrained MDPs with Bellman Distributional Certificatescited by 0