mlsanityData to 7 Oct 2026

Approximate Shielding of Atari Agents for Safe Exploration

arXiv 2304.11104 ↗First posted 21 Apr 2023cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Balancing exploration and conservatism in the constrained setting is an important problem if we are to use reinforcement learning for meaningful tasks in the real world. In this paper, we propose a principled algorithm for safe exploration based on the concept of shielding. Previous approaches to shielding assume access to a safety-relevant abstraction of the environment or a high-fidelity simulator. Instead, our work is based on latent shielding - another approach that leverages world models to verify policy roll-outs in the latent space of a learned dynamics model. Our novel algorithm builds on this previous work, using safety critics and other additional features to improve the stability and farsightedness of the algorithm. We demonstrate the effectiveness of our approach by running experiments on a small set of Atari games with state dependent safety labels. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations, and in some cases improves the speed of convergence and quality of the final agent.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

42 of 62 references.

References · 42

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
30 more
  1. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  2. 2017Safe Reinforcement Learning via Shieldingcited by 427
  3. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  5. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  6. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  7. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  8. 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 261
  9. 2020Projection-Based Constrained Policy Optimizationcited by 195
  10. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  11. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  12. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  13. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  14. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 60
  15. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  16. 2022Safe Reinforcement Learning via Shielding under Partial Observabilitycited by 49
  17. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  18. 2015Shield Synthesis: Runtime Enforcement for Reactive Systemscited by 41
  19. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  20. 2019Value constrained model-free continuous controlcited by 34
  21. 2019The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 25
  22. 2023DITTO: Offline Imitation Learning with World Modelscited by 20
  23. 2021Safe Continuous Control with Constrained Model-Based Policy Optimizationcited by 17
  24. 2023Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learningcited by 16
  25. 2021Shielding Atari Games with Bounded Presciencecited by 14
  26. 2021Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shieldingcited by 9
  27. 2022Reaching Through Latent Space: From Joint Statistics to Path Planning in Manipulationcited by 8
  28. 2022Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistakecited by 5
  29. 2022Shielding in Resource-Constrained Goal POMDPscited by 4
  30. 2023Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Reviewcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2024Leveraging Approximate Model-based Shielding for Probabilistic Safety Guarantees in Continuous Environmentscited by 5
  2. 2023Approximate Model-Based Shielding for Safe Reinforcement Learningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2023Model-based Dynamic Shielding for Safe and Efficient Multi-Agent Reinforcement Learning
  2. 2026Imagine to Ensure Safety in Hierarchical Reinforcement Learning
  3. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  4. 2026Explainably Safe Reinforcement Learning
  5. 2026Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning
  6. 2024Safe Exploration Using Bayesian World Models and Log-Barrier Optimization
  7. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  8. 2025Towards Safe Robot Foundation Models