# Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding

arXiv: https://arxiv.org/abs/2405.18180 · HTML: https://mlsanity.com/paper/2405.18180/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 28 May 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 30 of 30)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2310.12567/) (2023)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [Uncertainty-Aware Reinforcement Learning for Collision Avoidance](https://mlsanity.com/paper/1702.01182/) (2017)
- [Safe Reinforcement Learning via Curriculum Induction](https://mlsanity.com/paper/2006.12136/) (2020)
- [Constrained Variational Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2201.11927/) (2022)
- [Safe Multi-Agent Reinforcement Learning via Shielding](https://mlsanity.com/paper/2101.11196/) (2021)
- [Safe Reinforcement Learning via Shielding under Partial Observability](https://mlsanity.com/paper/2204.00755/) (2022)
- [Safe Reinforcement Learning via Probabilistic Logic Shields](https://mlsanity.com/paper/2303.03226/) (2023)
- [Online Shielding for Reinforcement Learning](https://mlsanity.com/paper/2212.01861/) (2022)
- [Dynamic Shielding for Reinforcement Learning in Black-Box Environments](https://mlsanity.com/paper/2207.13446/) (2022)
- [Safe Option-Critic: Learning Safety in the Option-Critic Architecture](https://mlsanity.com/paper/1807.08060/) (2018)
- [Reinforcement Learning with External Knowledge by using Logical Neural Networks](https://mlsanity.com/paper/2103.02363/) (2021)
- [Robust Uncertainty Quantification Using Conformalised Monte Carlo Prediction](https://mlsanity.com/paper/2308.09647/) (2023)
- [Automata Learning meets Shielding](https://mlsanity.com/paper/2212.01838/) (2022)
- [Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistake](https://mlsanity.com/paper/2202.09516/) (2022)
- [Approximate Model-Based Shielding for Safe Reinforcement Learning](https://mlsanity.com/paper/2308.00707/) (2023)

## Built on by (most cited first, 6 of 6)

- [Shields to Guarantee Probabilistic Safety in MDPs](https://mlsanity.com/paper/2605.10888/) (2026)
- [Robust Shielding for Safe Reinforcement Learning](https://mlsanity.com/paper/2606.00270/) (2026)
- [Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation](https://mlsanity.com/paper/2510.18478/) (2025)
- [Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2605.30576/) (2026)
- [Imagine to Ensure Safety in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2606.22509/) (2026)
- [Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning](https://mlsanity.com/paper/2608.19836/) (2026)

## Abstract (arXiv)

Empowering safe exploration of reinforcement learning (RL) agents during training is a critical challenge towards their deployment in many real-world scenarios. When prior knowledge of the domain or task is unavailable, training RL agents in unknown, black-box environments presents an even greater safety risk. We introduce ADVICE (Adaptive Shielding with a Contrastive Autoencoder), a novel post-shielding technique that distinguishes safe and unsafe features of state-action pairs during training, and uses this knowledge to protect the RL agent from executing actions that yield likely hazardous outcomes. Our comprehensive experimental evaluation against state-of-the-art safe RL exploration techniques shows that ADVICE significantly reduces safety violations (approx 50%) during training, with a competitive outcome reward compared to other techniques.