# Verifiably Safe Exploration for End-to-End Reinforcement Learning

arXiv: https://arxiv.org/abs/2007.01223 · HTML: https://mlsanity.com/paper/2007.01223/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 30 of 34)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Safe Reinforcement Learning via Probabilistic Logic Shields](https://mlsanity.com/paper/2303.03226/) (2023)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization](https://mlsanity.com/paper/2105.08881/) (2021)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [Online Shielding for Reinforcement Learning](https://mlsanity.com/paper/2212.01861/) (2022)
- [Provably Safe Deep Reinforcement Learning for Robotic Manipulation in Human Environments](https://mlsanity.com/paper/2205.06311/) (2022)
- [A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviors](https://mlsanity.com/paper/2302.13137/) (2023)
- [Dynamic Shielding for Reinforcement Learning in Black-Box Environments](https://mlsanity.com/paper/2207.13446/) (2022)
- [LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning](https://mlsanity.com/paper/2209.10341/) (2022)
- [Verified Probabilistic Policies for Deep Reinforcement Learning](https://mlsanity.com/paper/2201.03698/) (2022)
- [Ablation Study of How Run Time Assurance Impacts the Training and Performance of Reinforcement Learning Agents](https://mlsanity.com/paper/2207.04117/) (2022)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?](https://mlsanity.com/paper/2509.12833/) (2025)
- [MLNav: Learning to Safely Navigate on Martian Terrains](https://mlsanity.com/paper/2203.04563/) (2022)
- [Safety-Driven Deep Reinforcement Learning Framework for Cobots: A Sim2Real Approach](https://mlsanity.com/paper/2407.02231/) (2024)
- [Provably Optimal Reinforcement Learning under Safety Filtering](https://mlsanity.com/paper/2510.18082/) (2025)
- [Runtime Verification of Learning Properties for Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.09811/) (2023)
- [A Provable Approach for End-to-End Safe Reinforcement Learning](https://mlsanity.com/paper/2505.21852/) (2025)
- [Safe and Compliant Cross-Market Trade Execution via Constrained RL and Zero-Knowledge Audits](https://mlsanity.com/paper/2510.04952/) (2025)
- [X-Safe: Embodiment-Agnostic Safeguarding for Robot Arm Manipulation Policies via Action Masking](https://mlsanity.com/paper/2606.22278/) (2026)
- [Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies](https://mlsanity.com/paper/2405.01198/) (2024)
- [Think Smart, Act SMARL! Analyzing Probabilistic Logic Shields for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2411.04867/) (2024)
- [Predictive Safety Shield for Dyna-Q Reinforcement Learning](https://mlsanity.com/paper/2511.21531/) (2025)
- [GloCAL: Glocalized Curriculum-Aided Learning of Multiple Tasks with Application to Robotic Grasping](https://mlsanity.com/paper/2204.06835/) (2022)
- [Space Processor Computation Time Analysis for Reinforcement Learning and Run Time Assurance Control Policies](https://mlsanity.com/paper/2405.06771/) (2024)
- [Towards Online Safety Corrections for Robotic Manipulation Policies](https://mlsanity.com/paper/2409.08233/) (2024)
- [Progressive Safeguards for Safe and Model-Agnostic Reinforcement Learning](https://mlsanity.com/paper/2410.24096/) (2024)

## Abstract (arXiv)

Deploying deep reinforcement learning in safety-critical settings requires developing algorithms that obey hard constraints during exploration. This paper contributes a first approach toward enforcing formal safety constraints on end-to-end policies with visual inputs. Our approach draws on recent advances in object detection and automated reasoning for hybrid dynamical systems. The approach is evaluated on a novel benchmark that emphasizes the challenge of safely exploring in the presence of hard constraints. Our benchmark draws from several proposed problem sets for safe learning and includes problems that emphasize challenges such as reward signals that are not aligned with safety constraints. On each of these benchmark problems, our algorithm completely avoids unsafe behavior while remaining competitive at optimizing for as much reward as is safe. We also prove that our method of enforcing the safety constraints preserves all safe policies from the original environment.