# State-wise Constrained Policy Optimization

arXiv: https://arxiv.org/abs/2306.12594 · HTML: https://mlsanity.com/paper/2306.12594/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 27 of 27)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [Solving Linear Programs in the Current Matrix Multiplication Time](https://mlsanity.com/paper/1810.07896/) (2018)
- [Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/1802.06480/) (2018)
- [State-wise Safe Reinforcement Learning: A Survey](https://mlsanity.com/paper/2302.03122/) (2023)
- [Value constrained model-free continuous control](https://mlsanity.com/paper/1902.04623/) (2019)
- [Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety](https://mlsanity.com/paper/2105.10682/) (2021)
- [Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks](https://mlsanity.com/paper/2212.05727/) (2022)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)
- [Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Models](https://mlsanity.com/paper/2210.01041/) (2022)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [GUARD: A Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2305.13681/) (2023)
- [Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programming](https://mlsanity.com/paper/2209.06896/) (2022)
- [Safety Index Synthesis via Sum-of-Squares Programming](https://mlsanity.com/paper/2209.09134/) (2022)
- [Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithm](https://mlsanity.com/paper/1912.09095/) (2019)
- [Stochastic Variance Reduction for Deep Q-learning](https://mlsanity.com/paper/1905.08152/) (2019)
- [Experimental Evaluation of Human Motion Prediction: Toward Safe and Efficient Human Robot Collaboration](https://mlsanity.com/paper/2001.09550/) (2020)
- [Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimization](https://mlsanity.com/paper/2008.03826/) (2020)
- [SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving](https://mlsanity.com/paper/2206.08528/) (2022)
- [Provably Safe Tolerance Estimation for Robot Arms via Sum-of-Squares Programming](https://mlsanity.com/paper/2104.08896/) (2021)
- [Hybrid Task Constrained Planner for Robot Manipulator in Confined Environment](https://mlsanity.com/paper/2304.09260/) (2023)

## Built on by (most cited first, 9 of 9)

- [A Survey of Constraint Formulations in Safe Reinforcement Learning](https://mlsanity.com/paper/2402.02025/) (2024)
- [Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium](https://mlsanity.com/paper/2411.15036/) (2024)
- [Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.02754/) (2024)
- [Absolute Policy Optimization](https://mlsanity.com/paper/2310.13230/) (2023)
- [Safe and Near-Optimal Control with Online Dynamics Learning](https://mlsanity.com/paper/2509.16650/) (2025)
- [Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction](https://mlsanity.com/paper/2410.01212/) (2024)
- [Learning to Navigate Under Imperfect Perception: Conformalised Segmentation for Safe Reinforcement Learning](https://mlsanity.com/paper/2510.18485/) (2025)
- [A Safety-Constrained Reinforcement Learning Framework for Reliable Wireless Autonomy](https://mlsanity.com/paper/2602.13207/) (2026)
- [Dynamic-TD3: A Novel Algorithm for UAV Path Planning with Dynamic Obstacle Trajectory Prediction](https://mlsanity.com/paper/2605.00059/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) algorithms have shown tremendous success in simulation environments, but their application to real-world problems faces significant challenges, with safety being a major concern. In particular, enforcing state-wise constraints is essential for many challenging tasks such as autonomous driving and robot manipulation. However, existing safe RL algorithms under the framework of Constrained Markov Decision Process (CMDP) do not consider state-wise constraints. To address this gap, we propose State-wise Constrained Policy Optimization (SCPO), the first general-purpose policy search algorithm for state-wise constrained reinforcement learning. SCPO provides guarantees for state-wise constraint satisfaction in expectation. In particular, we introduce the framework of Maximum Markov Decision Process, and prove that the worst-case safety violation is bounded under SCPO. We demonstrate the effectiveness of our approach on training neural network policies for extensive robot locomotion tasks, where the agent must satisfy a variety of state-wise safety constraints. Our results show that SCPO significantly outperforms existing methods and can handle state-wise constraints in high-dimensional robotics tasks.