mlsanityData to 7 Oct 2026

Shielding Atari Games with Bounded Prescience

arXiv 2101.08153 ↗First posted 20 Jan 2021cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (DRL) is applied in safety-critical domains such as robotics and autonomous driving. It achieves superhuman abilities in many tasks, however whether DRL agents can be shown to act safely is an open problem. Atari games are a simple yet challenging exemplar for evaluating the safety of DRL agents and feature a diverse portfolio of game mechanics. The safety of neural agents has been studied before using methods that either require a model of the system dynamics or an abstraction; unfortunately, these are unsuitable to Atari games because their low-level dynamics are complex and hidden inside their emulator. We present the first exact method for analysing and ensuring the safety of DRL agents for Atari games. Our method only requires access to the emulator. First, we give a set of 43 properties that characterise "safe behaviour" for 30 games. Second, we develop a method for exploring all traces induced by an agent and a game and consider a variety of sources of game non-determinism. We observe that the best available DRL agents reliably satisfy only very few properties; several critical properties are violated by all agents. Finally, we propose a countermeasure that combines a bounded explicit-state exploration with shielding. We demonstrate that our method improves the safety of all agents over multiple properties.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 29 of its 62 references that are in the map. First posted on arXiv 20 Jan 2021.

Builds on

29 of 62 references.

References · 29

  1. 2013Intriguing properties of neural networkscited by 5,114
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2016OpenAI Gymcited by 1,684
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  8. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  9. 2017Safe Reinforcement Learning via Shieldingcited by 427
  10. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  11. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  12. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
17 more
  1. 2016Safety Verification of Deep Neural Networkscited by 262
  2. 2018Distributed Prioritized Experience Replaycited by 260
  3. 2020NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systemscited by 128
  4. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  5. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  6. 2020Verification of Deep Convolutional Neural Networks Using ImageStarscited by 75
  7. 2019Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamicscited by 45
  8. 2020Cautious Reinforcement Learning with Logical Constraintscited by 44
  9. 2019Certified Reinforcement Learning with Logic Guidancecited by 35
  10. 2019MAMPS: Safe Multi-Agent Reinforcement Learning via Model Predictive Shieldingcited by 17
  11. 2020Adaptive Shielding under Uncertaintycited by 17
  12. 2019Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing fieldcited by 14
  13. 2014FAUST2: Formal Abstractions of Uncountable-STate STochastic processescited by 12
  14. 2020Online Shielding for Stochastic Systemscited by 11
  15. 2020Formal Methods with a Touch of Magiccited by 10
  16. 2020It's Time to Play Safe: Shield Synthesis for Timed Systemscited by 4
  17. 2019Simplifying Neural Networks using Formal Verificationcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 5; 2024: 3.

14 papers.

Sort:

  1. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 41
  2. 2022Learning Control Policies for Stochastic Systems with Reach-avoid Guaranteescited by 21
  3. 2022Online Shielding for Reinforcement Learningcited by 19
  4. 2022Correct-by-Construction Runtime Enforcement in AI -- A Surveycited by 10
  5. 2021Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shieldingcited by 9
  6. 2022Automata Learning meets Shieldingcited by 9
  7. 2022Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistakecited by 5
  8. 2024Leveraging Approximate Model-based Shielding for Probabilistic Safety Guarantees in Continuous Environmentscited by 5
  9. 2023Approximate Model-Based Shielding for Safe Reinforcement Learningcited by 4
  10. 2023Approximate Shielding of Atari Agents for Safe Explorationcited by 2
  11. 2024Composing Reinforcement Learning Policies, with Formal Guaranteescited by 2
  12. 2023Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesiscited by 1
2 more
  1. 2023Experiments with Detecting and Mitigating AI Deceptioncited by 0
  2. 2024Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learningcited by 0