mlsanityData to 7 Oct 2026

Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction

arXiv 2410.01212 ↗First posted 2 Oct 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Enforcing state-wise safety constraints is critical for the application of reinforcement learning (RL) in real-world problems, such as autonomous driving and robot manipulation. However, existing safe RL methods only enforce state-wise constraints in expectation or enforce hard state-wise constraints with strong assumptions. The former does not exclude the probability of safety violations, while the latter is impractical. Our insight is that although it is intractable to guarantee hard state-wise constraints in a model-free setting, we can enforce state-wise safety with high probability while excluding strong assumptions. To accomplish the goal, we propose Absolute State-wise Constrained Policy Optimization (ASCPO), a novel general-purpose policy search algorithm that guarantees high-probability state-wise constraint satisfaction for stochastic systems. We demonstrate the effectiveness of our approach by training neural network policies for extensive robot locomotion tasks, where the agent must adhere to various state-wise safety constraints. Our results show that ASCPO significantly outperforms existing methods in handling state-wise constraints across challenging continuous control tasks, highlighting its potential for real-world applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

47 of 60 references.

References · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  5. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  6. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  7. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  8. 2018Safe Exploration in Continuous Action Spacescited by 237
  9. 2020Projection-Based Constrained Policy Optimizationcited by 195
  10. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  11. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  12. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
35 more
  1. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  2. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  3. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  4. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 109
  5. 2020Conservative Safety Critics for Explorationcited by 107
  6. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 64
  7. 2022Reachability Constrained Reinforcement Learningcited by 52
  8. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  9. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  10. 2019Value constrained model-free continuous controlcited by 34
  11. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  12. 2020Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Controlcited by 28
  13. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 27
  14. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18
  15. 2020ShieldNN: A Provably Safe NN Filter for Unsafe NN Controllerscited by 17
  16. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 16
  17. 2023A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraintscited by 15
  18. 2022Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Modelscited by 14
  19. 2023GUARD: A Safe Reinforcement Learning Benchmarkcited by 13
  20. 2022Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programmingcited by 11
  21. 2022Safety Index Synthesis via Sum-of-Squares Programmingcited by 11
  22. 2023State-wise Constrained Policy Optimizationcited by 9
  23. 2019Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithmcited by 8
  24. 2019Stochastic Variance Reduction for Deep Q-learningcited by 7
  25. 2020Experimental Evaluation of Human Motion Prediction: Toward Safe and Efficient Human Robot Collaborationcited by 7
  26. 2020Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimizationcited by 7
  27. 2021Provably Safe Tolerance Estimation for Robot Arms via Sum-of-Squares Programmingcited by 6
  28. 2022SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Drivingcited by 6
  29. 2023Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimizationcited by 4
  30. 2023Learning Predictive Safety Filter via Decomposition of Robust Invariant Setcited by 3
  31. 2024Real-time Safety Index Adaptation for Parameter-varying Systems via Determinant Gradient Ascendcited by 3
  32. 2024Implicit Safe Set Algorithm for Provably Safe Reinforcement Learningcited by 3
  33. 2023Safety Index Synthesis with State-dependent Control Spacecited by 2
  34. 2023Absolute Policy Optimizationcited by 2
  35. 2024Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skillscited by 2

Built on by

Not cited yet.