mlsanityData to 7 Oct 2026

SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization

arXiv 2311.00880 ↗First posted 1 Nov 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Incorporating safety is an essential prerequisite for broadening the practical applications of reinforcement learning in real-world scenarios. To tackle this challenge, Constrained Markov Decision Processes (CMDPs) are leveraged, which introduce a distinct cost function representing safety violations. In CMDPs' settings, Lagrangian relaxation technique has been employed in previous algorithms to convert constrained optimization problems into unconstrained dual problems. However, these algorithms may inaccurately predict unsafe behavior, resulting in instability while learning the Lagrange multiplier. This study introduces a novel safe reinforcement learning algorithm, Safety Critic Policy Optimization (SCPO). In this study, we define the safety critic, a mechanism that nullifies rewards obtained through violating safety constraints. Furthermore, our theoretical analysis indicates that the proposed algorithm can automatically balance the trade-off between adhering to safety constraints and maximizing rewards. The effectiveness of the SCPO algorithm is empirically validated by benchmarking it against strong baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 30 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

17 of 30 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  6. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  7. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  8. 2020Projection-Based Constrained Policy Optimizationcited by 195
  9. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  10. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  11. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 64
  12. 2020Cautious Reinforcement Learning with Logical Constraintscited by 44
5 more
  1. 2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 37
  2. 2020Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationshipscited by 36
  3. 2022i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loopscited by 30
  4. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  5. 2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 13

Built on by

Not cited yet.