# Online Shielding for Reinforcement Learning

arXiv: https://arxiv.org/abs/2212.01861 · HTML: https://mlsanity.com/paper/2212.01861/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 4 Dec 2022.

## Builds on (most cited first, 22 of 22)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Omega-Regular Objectives in Model-Free Reinforcement Learning](https://mlsanity.com/paper/1810.00950/) (2018)
- [Safe Multi-Agent Reinforcement Learning via Shielding](https://mlsanity.com/paper/2101.11196/) (2021)
- [Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamics](https://mlsanity.com/paper/1910.10885/) (2019)
- [Cautious Reinforcement Learning with Logical Constraints](https://mlsanity.com/paper/2002.12156/) (2020)
- [Shield Synthesis: Runtime Enforcement for Reactive Systems](https://mlsanity.com/paper/1501.02573/) (2015)
- [Learning Robotic Manipulation through Visual Planning and Acting](https://mlsanity.com/paper/1905.04411/) (2019)
- [Certified Reinforcement Learning with Logic Guidance](https://mlsanity.com/paper/1902.00778/) (2019)
- [A storm is Coming: A Modern Probabilistic Model Checker](https://mlsanity.com/paper/1702.04311/) (2017)
- [Verifiably Safe Exploration for End-to-End Reinforcement Learning](https://mlsanity.com/paper/2007.01223/) (2020)
- [MAMPS: Safe Multi-Agent Reinforcement Learning via Model Predictive Shielding](https://mlsanity.com/paper/1910.12639/) (2019)
- [Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigation](https://mlsanity.com/paper/1801.09627/) (2018)
- [Adaptive Shielding under Uncertainty](https://mlsanity.com/paper/2010.03842/) (2020)
- [Verifiably Safe Off-Model Reinforcement Learning](https://mlsanity.com/paper/1902.05632/) (2019)
- [Shielding Atari Games with Bounded Prescience](https://mlsanity.com/paper/2101.08153/) (2021)
- [TEMPEST -- Synthesis Tool for Reactive Systems and Shields in Probabilistic Environments](https://mlsanity.com/paper/2105.12588/) (2021)
- [Online Shielding for Stochastic Systems](https://mlsanity.com/paper/2012.09539/) (2020)
- [L*-Based Learning of Markov Decision Processes (Extended Version)](https://mlsanity.com/paper/1906.12239/) (2019)

## Built on by (most cited first, 22 of 22)

- [Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.13863/) (2024)
- [SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2308.02594/) (2023)
- [Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward](https://mlsanity.com/paper/2404.08517/) (2024)
- [Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding](https://mlsanity.com/paper/2405.18180/) (2024)
- [Breaking Up with Normatively Monolithic Agency with GRACE: A Reason-Based Neuro-Symbolic Architecture for Safe and Ethical AI Alignment](https://mlsanity.com/paper/2601.10520/) (2026)
- [CommonPower: A Framework for Safe Data-Driven Smart Grid Control](https://mlsanity.com/paper/2406.03231/) (2024)
- [Conformal Safety Shielding for Imperfect-Perception Agents](https://mlsanity.com/paper/2506.17275/) (2025)
- [HypRL: Reinforcement Learning of Control Policies for Hyperproperties](https://mlsanity.com/paper/2504.04675/) (2025)
- [Automaton Constrained Q-Learning](https://mlsanity.com/paper/2510.05061/) (2025)
- [Shields to Guarantee Probabilistic Safety in MDPs](https://mlsanity.com/paper/2605.10888/) (2026)
- [Robust Shielding for Safe Reinforcement Learning](https://mlsanity.com/paper/2606.00270/) (2026)
- [SPoRt -- Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL](https://mlsanity.com/paper/2504.06386/) (2025)
- [Follow the STARs: Dynamic ω-Regular Shielding of Learned Policies](https://mlsanity.com/paper/2505.14689/) (2025)
- [Conformal Signal Temporal Logic for Robust Reinforcement Learning Control: A Case Study](https://mlsanity.com/paper/2602.14322/) (2026)
- [Constrained Exploration in Reinforcement Learning with Optimality Preservation](https://mlsanity.com/paper/2304.03104/) (2023)
- [Human-aligned Safe Reinforcement Learning for Highway On-Ramp Merging in Dense Traffic](https://mlsanity.com/paper/2503.02624/) (2025)
- [Risk-Aware Safe Reinforcement Learning for Control of Stochastic Linear Systems](https://mlsanity.com/paper/2505.09734/) (2025)
- [Efficient Dynamic Shielding for Parametric Safety Specifications](https://mlsanity.com/paper/2505.22104/) (2025)
- [SORA-ATMAS: Adaptive Trust Management and Multi-LLM Aligned Governance for Future Smart Cities](https://mlsanity.com/paper/2510.19327/) (2025)
- [Adaptive GR(1) Specification Repair for Liveness-Preserving Shielding in Reinforcement Learning](https://mlsanity.com/paper/2511.02605/) (2025)
- [COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Management](https://mlsanity.com/paper/2603.02396/) (2026)
- [Data-Driven Synthesis of Probabilistic Controlled Invariant Sets for Linear MDPs](https://mlsanity.com/paper/2604.02727/) (2026)

## Abstract (arXiv)

Besides the recent impressive results on reinforcement learning (RL), safety is still one of the major research challenges in RL. RL is a machine-learning approach to determine near-optimal policies in Markov decision processes (MDPs). In this paper, we consider the setting where the safety-relevant fragment of the MDP together with a temporal logic safety specification is given and many safety violations can be avoided by planning ahead a short time into the future. We propose an approach for online safety shielding of RL agents. During runtime, the shield analyses the safety of each available action. For any action, the shield computes the maximal probability to not violate the safety specification within the next k steps when executing this action. Based on this probability and a given threshold, the shield decides whether to block an action from the agent. Existing offline shielding approaches compute exhaustively the safety of all state-action combinations ahead of time, resulting in huge computation times and large memory consumption. The intuition behind online shielding is to compute at runtime the set of all states that could be reached in the near future. For each of these states, the safety of all available actions is analysed and used for shielding as soon as one of the considered states is reached. Our approach is well suited for high-level planning problems where the time between decisions can be used for safety computations and it is sustainable for the agent to wait until these computations are finished. For our evaluation, we selected a 2-player version of the classical computer game SNAKE. The game represents a high-level planning problem that requires fast decisions and the multiplayer setting induces a large state space, which is computationally expensive to analyse exhaustively.