# Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve Optimality?

arXiv: https://arxiv.org/abs/2004.00915 · HTML: https://mlsanity.com/paper/2004.00915/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2020.

## Built on by (most cited first, 30 of 34)

- [Nonlinear Invariant Risk Minimization: A Causal Approach](https://mlsanity.com/paper/2102.12353/) (2021)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [Contingency-constrained economic dispatch with safe reinforcement learning](https://mlsanity.com/paper/2205.06212/) (2022)
- [Optimization of the Model Predictive Control Meta-Parameters Through Reinforcement Learning](https://mlsanity.com/paper/2111.04146/) (2021)
- [Anytime-Constrained Reinforcement Learning](https://mlsanity.com/paper/2311.05511/) (2023)
- [Stability-Constrained Markov Decision Processes Using MPC](https://mlsanity.com/paper/2102.01383/) (2021)
- [Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time](https://mlsanity.com/paper/2405.14183/) (2024)
- [Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?](https://mlsanity.com/paper/2509.12833/) (2025)
- [From Demonstrations to Safe Deployment: Path-Consistent Safety Filtering for Diffusion Policies](https://mlsanity.com/paper/2511.06385/) (2025)
- [CommonPower: A Framework for Safe Data-Driven Smart Grid Control](https://mlsanity.com/paper/2406.03231/) (2024)
- [Towards graph neural networks for provably solving convex optimization problems](https://mlsanity.com/paper/2502.02446/) (2025)
- [Provably Optimal Reinforcement Learning under Safety Filtering](https://mlsanity.com/paper/2510.18082/) (2025)
- [Control invariant set enhanced safe reinforcement learning: improved sampling efficiency, guaranteed stability and robustness](https://mlsanity.com/paper/2305.15602/) (2023)
- [Safe Reinforcement Learning for Strategic Bidding of Virtual Power Plants in Day-Ahead Markets](https://mlsanity.com/paper/2307.05812/) (2023)
- [A Constraint Enforcement Deep Reinforcement Learning Framework for Optimal Energy Storage Systems Dispatch](https://mlsanity.com/paper/2307.14304/) (2023)
- [Safe and Compliant Cross-Market Trade Execution via Constrained RL and Zero-Knowledge Audits](https://mlsanity.com/paper/2510.04952/) (2025)
- [X-Safe: Embodiment-Agnostic Safeguarding for Robot Arm Manipulation Policies via Action Masking](https://mlsanity.com/paper/2606.22278/) (2026)
- [Control invariant set enhanced reinforcement learning for process control: improved sampling efficiency and guaranteed stability](https://mlsanity.com/paper/2304.05509/) (2023)
- [Modeling Risk in Reinforcement Learning: A Literature Mapping](https://mlsanity.com/paper/2312.05231/) (2023)
- [Adaptive Safe Reinforcement Learning-Enabled Optimization of Battery Fast-Charging Protocols](https://mlsanity.com/paper/2406.12309/) (2024)
- [Polynomial-Time Approximability of Constrained Reinforcement Learning](https://mlsanity.com/paper/2502.07764/) (2025)
- [Neural Lyapunov Function Approximation with Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2503.15629/) (2025)
- [Leveraging Analytic Gradients in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2506.01665/) (2025)
- [Safe Control and Learning Using Generalized Action Governor](https://mlsanity.com/paper/2211.12628/) (2022)
- [Computationally Efficient Reinforcement Learning: Targeted Exploration leveraging Simple Rules](https://mlsanity.com/paper/2211.16691/) (2022)
- [Convergent NMPC-based Reinforcement Learning Using Deep Expected Sarsa and Nonlinear Temporal Difference Learning](https://mlsanity.com/paper/2502.04925/) (2025)
- [Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents](https://mlsanity.com/paper/2507.13491/) (2025)

## Abstract (arXiv)

For all its successes, Reinforcement Learning (RL) still struggles to deliver formal guarantees on the closed-loop behavior of the learned policy. Among other things, guaranteeing the safety of RL with respect to safety-critical systems is a very active research topic. Some recent contributions propose to rely on projections of the inputs delivered by the learned policy into a safe set, ensuring that the system safety is never jeopardized. Unfortunately, it is unclear whether this operation can be performed without disrupting the learning process. This paper addresses this issue. The problem is analysed in the context of Q-learning and policy gradient techniques. We show that the projection approach is generally disruptive in the context of Q-learning though a simple alternative solves the issue, while simple corrections can be used in the context of policy gradient methods in order to ensure that the policy gradients are unbiased. The proposed results extend to safe projections based on robust MPC techniques.