# State-wise Safe Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2302.03122 · HTML: https://mlsanity.com/paper/2302.03122/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [On Kernelized Multi-armed Bandits](https://mlsanity.com/paper/1704.00445/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Safe Reinforcement Learning in Constrained Markov Decision Processes](https://mlsanity.com/paper/2008.06626/) (2020)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/1802.06480/) (2018)
- [Value constrained model-free continuous control](https://mlsanity.com/paper/1902.04623/) (2019)
- [Reachability-based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Control](https://mlsanity.com/paper/2011.08421/) (2020)
- [Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.07695/) (2021)
- [Safe Control with Neural Network Dynamic Models](https://mlsanity.com/paper/2110.01110/) (2021)
- [Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks](https://mlsanity.com/paper/2212.05727/) (2022)
- [Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions](https://mlsanity.com/paper/1908.01883/) (2019)
- [ShieldNN: A Provably Safe NN Filter for Unsafe NN Controllers](https://mlsanity.com/paper/2006.09564/) (2020)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)
- [Probabilistic Safeguard for Reinforcement Learning Using Safety Index Guided Gaussian Process Models](https://mlsanity.com/paper/2210.01041/) (2022)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [Persistently Feasible Robust Safe Control by Safety Index Synthesis and Convex Semi-Infinite Programming](https://mlsanity.com/paper/2209.06896/) (2022)
- [Safety Index Synthesis via Sum-of-Squares Programming](https://mlsanity.com/paper/2209.09134/) (2022)
- [Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithm](https://mlsanity.com/paper/1912.09095/) (2019)
- [Stochastic Variance Reduction for Deep Q-learning](https://mlsanity.com/paper/1905.08152/) (2019)
- [Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimization](https://mlsanity.com/paper/2008.03826/) (2020)

## Built on by (most cited first, 30 of 55)

- [Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion](https://mlsanity.com/paper/2401.17583/) (2024)
- [Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model](https://mlsanity.com/paper/2401.10700/) (2024)
- [A Survey of Constraint Formulations in Safe Reinforcement Learning](https://mlsanity.com/paper/2402.02025/) (2024)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2407.09645/) (2024)
- [Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.13863/) (2024)
- [A Survey on Imitation Learning for Contact-Rich Tasks in Robotics](https://mlsanity.com/paper/2506.13498/) (2025)
- [GUARD: A Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2305.13681/) (2023)
- [Verification of Neural Control Barrier Functions with Symbolic Derivative Bounds Propagation](https://mlsanity.com/paper/2410.16281/) (2024)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [State-wise Constrained Policy Optimization](https://mlsanity.com/paper/2306.12594/) (2023)
- [Safe Deep Policy Adaptation](https://mlsanity.com/paper/2310.08602/) (2023)
- [Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation](https://mlsanity.com/paper/2405.20860/) (2024)
- [Multi-Agent Reinforcement Learning Guided by Signal Temporal Logic Specifications](https://mlsanity.com/paper/2306.06808/) (2023)
- [Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium](https://mlsanity.com/paper/2411.15036/) (2024)
- [Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery](https://mlsanity.com/paper/2306.13944/) (2023)
- [Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time](https://mlsanity.com/paper/2405.14183/) (2024)
- [Action Mapping for Reinforcement Learning in Continuous Environments with Constraints](https://mlsanity.com/paper/2412.04327/) (2024)
- [Emergent Risk Awareness in Rational Agents under Resource Constraints](https://mlsanity.com/paper/2505.23436/) (2025)
- [Robust Safe Reinforcement Learning under Adversarial Disturbances](https://mlsanity.com/paper/2310.07207/) (2023)
- [Learning Predictive Safety Filter via Decomposition of Robust Invariant Set](https://mlsanity.com/paper/2311.06769/) (2023)
- [Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.02754/) (2024)
- [Speedup Patch: Learning a Plug-and-Play Policy to Accelerate Embodied Manipulation](https://mlsanity.com/paper/2603.20658/) (2026)
- [Absolute Policy Optimization](https://mlsanity.com/paper/2310.13230/) (2023)
- [Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skills](https://mlsanity.com/paper/2405.11380/) (2024)
- [Combining Deep Reinforcement Learning with a Jerk-Bounded Trajectory Generator for Kinematically Constrained Motion Planning](https://mlsanity.com/paper/2410.20907/) (2024)
- [Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning](https://mlsanity.com/paper/2502.10431/) (2025)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signals](https://mlsanity.com/paper/2510.14503/) (2025)
- [EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems](https://mlsanity.com/paper/2604.11174/) (2026)

## Abstract (arXiv)

Despite the tremendous success of Reinforcement Learning (RL) algorithms in simulation environments, applying RL to real-world applications still faces many challenges. A major concern is safety, in another word, constraint satisfaction. State-wise constraints are one of the most common constraints in real-world applications and one of the most challenging constraints in Safe RL. Enforcing state-wise constraints is necessary and essential to many challenging tasks such as autonomous driving, robot manipulation. This paper provides a comprehensive review of existing approaches that address state-wise constraints in RL. Under the framework of State-wise Constrained Markov Decision Process (SCMDP), we will discuss the connections, differences, and trade-offs of existing approaches in terms of (i) safety guarantee and scalability, (ii) safety and reward performance, and (iii) safety after convergence and during training. We also summarize limitations of current methods and discuss potential future directions.