mlsanityData to 14 Oct 2026

State-wise Constrained Policy Optimization

arXiv 2306.12594 ↗First posted 21 Jun 2023cs.LG cs.ROcited by 9

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) algorithms have shown tremendous success in simulation environments, but their application to real-world problems faces significant challenges, with safety being a major concern. In particular, enforcing state-wise constraints is essential for many challenging tasks such as autonomous driving and robot manipulation. However, existing safe RL algorithms under the framework of Constrained Markov Decision Process (CMDP) do not consider state-wise constraints. To address this gap, we propose State-wise Constrained Policy Optimization (SCPO), the first general-purpose policy search algorithm for state-wise constrained reinforcement learning. SCPO provides guarantees for state-wise constraint satisfaction in expectation. In particular, we introduce the framework of Maximum Markov Decision Process, and prove that the worst-case safety violation is bounded under SCPO. We demonstrate the effectiveness of our approach on training neural network policies for extensive robot locomotion tasks, where the agent must satisfy a variety of state-wise safety constraints. Our results show that SCPO significantly outperforms existing methods and can handle state-wise constraints in high-dimensional robotics tasks.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

27 of 41 references.

References · 27

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2018Safe Exploration in Continuous Action Spacescited by 247
  4. 2020Projection-Based Constrained Policy Optimizationcited by 203
  5. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  6. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  7. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  8. 2020Conservative Safety Critics for Explorationcited by 113
  9. 2018Solving Linear Programs in the Current Matrix Multiplication Timecited by 100
  10. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  11. 2023State-wise Safe Reinforcement Learning: A Surveycited by 55
  12. 2019Value constrained model-free continuous controlcited by 34
15 more
  1. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  2. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 22
  3. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 17
  4. 2022Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Modelscited by 15
  5. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 13
  6. 2023GUARD: A Safe Reinforcement Learning Benchmarkcited by 13
  7. 2022Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programmingcited by 12
  8. 2022Safety Index Synthesis via Sum-of-Squares Programmingcited by 12
  9. 2019Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithmcited by 8
  10. 2019Stochastic Variance Reduction for Deep Q-learningcited by 7
  11. 2020Experimental Evaluation of Human Motion Prediction: Toward Safe and Efficient Human Robot Collaborationcited by 7
  12. 2020Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimizationcited by 7
  13. 2022SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Drivingcited by 7
  14. 2021Provably Safe Tolerance Estimation for Robot Arms via Sum-of-Squares Programmingcited by 6
  15. 2023Hybrid Task Constrained Planner for Robot Manipulator in Confined Environmentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 2.

9 papers.

  1. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  2. 2024Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibriumcited by 6
  3. 2024Implicit Safe Set Algorithm for Provably Safe Reinforcement Learningcited by 3
  4. 2023Absolute Policy Optimizationcited by 2
  5. 2025Safe and Near-Optimal Control with Online Dynamics Learningcited by 1
  6. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfactioncited by 0
  7. 2025Learning to Navigate Under Imperfect Perception: Conformalised Segmentation for Safe Reinforcement Learningcited by 0
  8. 2026A Safety-Constrained Reinforcement Learning Framework for Reliable Wireless Autonomycited by 0
  9. 2026Dynamic-TD3: A Novel Algorithm for UAV Path Planning with Dynamic Obstacle Trajectory Predictioncited by 0