# Safe Reinforcement Learning via Probabilistic Logic Shields

arXiv: https://arxiv.org/abs/2303.03226 · HTML: https://mlsanity.com/paper/2303.03226/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 77 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [A Semantic Loss Function for Deep Learning with Symbolic Knowledge](https://mlsanity.com/paper/1711.11157/) (2017)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Semantic Probabilistic Layers for Neuro-Symbolic Learning](https://mlsanity.com/paper/2206.00426/) (2022)
- [A Differential Approach to Inference in Bayesian Networks](https://mlsanity.com/paper/1301.3847/) (2013)
- [Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guarantees](https://mlsanity.com/paper/1909.05304/) (2019)
- [Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications](https://mlsanity.com/paper/1807.06333/) (2018)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Safe Reinforcement Learning via Shielding under Partial Observability](https://mlsanity.com/paper/2204.00755/) (2022)
- [QMDP-Net: Deep Learning for Planning under Partial Observability](https://mlsanity.com/paper/1703.06692/) (2017)
- [Neurosymbolic Reinforcement Learning with Formally Verified Exploration](https://mlsanity.com/paper/2009.12612/) (2020)
- [Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamics](https://mlsanity.com/paper/1910.10885/) (2019)
- [Self-Supervised Online Reward Shaping in Sparse-Reward Environments](https://mlsanity.com/paper/2103.04529/) (2021)
- [Verifiably Safe Exploration for End-to-End Reinforcement Learning](https://mlsanity.com/paper/2007.01223/) (2020)
- [Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks](https://mlsanity.com/paper/2007.01498/) (2020)
- [Deep Inverse Q-learning with Constraints](https://mlsanity.com/paper/2008.01712/) (2020)
- [Enforcing Almost-Sure Reachability in POMDPs](https://mlsanity.com/paper/2007.00085/) (2020)
- [Shielding Atari Games with Bounded Prescience](https://mlsanity.com/paper/2101.08153/) (2021)
- [Reinforcement Learning with External Knowledge by using Logical Neural Networks](https://mlsanity.com/paper/2103.02363/) (2021)
- [Parameterized Verification of Asynchronous Shared-Memory Systems](https://mlsanity.com/paper/1304.1185/) (2013)

## Built on by (most cited first, 30 of 42)

- [KLay: Accelerating Arithmetic Circuits for Neurosymbolic AI](https://mlsanity.com/paper/2410.11415/) (2024)
- [Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.13863/) (2024)
- [Probabilistic Shielding for Safe Reinforcement Learning](https://mlsanity.com/paper/2503.07671/) (2025)
- [Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture](https://mlsanity.com/paper/2502.15620/) (2025)
- [NeSyA: Neurosymbolic Automata](https://mlsanity.com/paper/2412.07331/) (2024)
- [Building Intelligent Agents with Neuro-Symbolic Concepts](https://mlsanity.com/paper/2505.06191/) (2025)
- [Compositional Shielding and Reinforcement Learning for Multi-Agent Systems](https://mlsanity.com/paper/2410.10460/) (2024)
- [Defining neurosymbolic AI](https://mlsanity.com/paper/2507.11127/) (2025)
- [Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2307.01316/) (2023)
- [Semirings for Probabilistic and Neuro-Symbolic Logic Programming](https://mlsanity.com/paper/2402.13782/) (2024)
- [Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding](https://mlsanity.com/paper/2405.18180/) (2024)
- [Leveraging Approximate Model-based Shielding for Probabilistic Safety Guarantees in Continuous Environments](https://mlsanity.com/paper/2402.00816/) (2024)
- [Realizable Continuous-Space Shields for Safe Reinforcement Learning](https://mlsanity.com/paper/2410.02038/) (2024)
- [Learning-Based Shielding for Safe Autonomy under Unknown Dynamics](https://mlsanity.com/paper/2410.07359/) (2024)
- [Provably Optimal Reinforcement Learning under Safety Filtering](https://mlsanity.com/paper/2510.18082/) (2025)
- [The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy](https://mlsanity.com/paper/2510.26752/) (2025)
- [Composing Reinforcement Learning Policies, with Formal Guarantees](https://mlsanity.com/paper/2402.13785/) (2024)
- [Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning](https://mlsanity.com/paper/2411.07700/) (2024)
- [Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shifts](https://mlsanity.com/paper/2506.11033/) (2025)
- [Neuro-Symbolic Artificial Intelligence: A Task-Directed Survey in the Black-Box Models Era](https://mlsanity.com/paper/2603.03177/) (2026)
- [Shields to Guarantee Probabilistic Safety in MDPs](https://mlsanity.com/paper/2605.10888/) (2026)
- [Robust Shielding for Safe Reinforcement Learning](https://mlsanity.com/paper/2606.00270/) (2026)
- [Integrating Symbolic Reasoning into Neural Generative Models for Design Generation](https://mlsanity.com/paper/2310.09383/) (2023)
- [Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies](https://mlsanity.com/paper/2405.01198/) (2024)
- [Think Smart, Act SMARL! Analyzing Probabilistic Logic Shields for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2411.04867/) (2024)
- [Follow the STARs: Dynamic ω-Regular Shielding of Learned Policies](https://mlsanity.com/paper/2505.14689/) (2025)
- [Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed](https://mlsanity.com/paper/2601.21094/) (2026)
- [Neuro-symbolic Action Masking for Deep Reinforcement Learning](https://mlsanity.com/paper/2602.10598/) (2026)
- [Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment](https://mlsanity.com/paper/2605.18672/) (2026)
- [Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning](https://mlsanity.com/paper/2605.25267/) (2026)

## Abstract (arXiv)

Safe Reinforcement learning (Safe RL) aims at learning optimal policies while staying safe. A popular solution to Safe RL is shielding, which uses a logical safety specification to prevent an RL agent from taking unsafe actions. However, traditional shielding techniques are difficult to integrate with continuous, end-to-end deep RL methods. To this end, we introduce Probabilistic Logic Policy Gradient (PLPG). PLPG is a model-based Safe RL technique that uses probabilistic logic programming to model logical safety constraints as differentiable functions. Therefore, PLPG can be seamlessly applied to any policy gradient algorithm while still providing the same convergence guarantees. In our experiments, we show that PLPG learns safer and more rewarding policies compared to other state-of-the-art shielding techniques.