# Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimization

arXiv: https://arxiv.org/abs/2308.13140 · HTML: https://mlsanity.com/paper/2308.13140/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2023.

## Built on by (most cited first, 4 of 4)

- [Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents](https://mlsanity.com/paper/2410.11671/) (2024)
- [Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.02754/) (2024)
- [Absolute Policy Optimization](https://mlsanity.com/paper/2310.13230/) (2023)
- [Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction](https://mlsanity.com/paper/2410.01212/) (2024)

## Abstract (arXiv)

Deep reinforcement learning (RL) excels in various control tasks, yet the absence of safety guarantees hampers its real-world applicability. In particular, explorations during learning usually results in safety violations, while the RL agent learns from those mistakes. On the other hand, safe control techniques ensure persistent safety satisfaction but demand strong priors on system dynamics, which is usually hard to obtain in practice. To address these problems, we present Safe Set Guided State-wise Constrained Policy Optimization (S-3PO), a pioneering algorithm generating state-wise safe optimal policies with zero training violations, i.e., learning without mistakes. S-3PO first employs a safety-oriented monitor with black-box dynamics to ensure safe exploration. It then enforces an "imaginary" cost for the RL agent to converge to optimal behaviors within safety constraints. S-3PO outperforms existing methods in high-dimensional robotics tasks, managing state-wise constraints with zero training violation. This innovation marks a significant stride towards real-world safe RL deployment.