mlsanityData to 7 Oct 2026

AI Safety Gridworlds

arXiv 1711.09883 ↗First posted 27 Nov 2017cs.LG cs.AIcited by 108

Not on the map. Why?

Abstract (arXiv)

We present a suite of reinforcement learning environments illustrating various safety properties of intelligent agents. These problems include safe interruptibility, avoiding side effects, absent supervisor, reward gaming, safe exploration, as well as robustness to self-modification, distributional shift, and adversaries. To measure compliance with the intended safe behavior, we equip each environment with a performance function that is hidden from the agent. This allows us to categorize AI safety problems into robustness and specification problems, depending on whether the performance function corresponds to the observed reward function. We evaluate A2C and Rainbow, two recent deep reinforcement learning agents, on our environments and show that they are not able to solve them satisfactorily.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 2; 2020: 1; 2021: 19; 2022: 12; 2023: 21; 2024: 7; 2025: 17; 2026: 27.

108 papers.

Sort:

By area:

  1. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  4. 2023Large Language Model Alignment: A Surveycited by 241
  5. 2021Alignment of Language Agentscited by 167
  6. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  7. 2021Consequences of Misaligned AIcited by 79
  8. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  9. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  10. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  11. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  12. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
38 more
  1. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 43
  2. 2021safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Roboticscited by 42
  3. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  4. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  5. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  6. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  7. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
  8. 2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 22
  9. 2023Interactive AI Alignment: Specification, Process, and Evaluation Alignmentcited by 22
  10. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  11. 2023Concrete Problems in AI Safety, Revisitedcited by 21
  12. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  13. 2021Concave Utility Reinforcement Learning with Zero-Constraint Violationscited by 18
  14. 2021There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learningcited by 17
  15. 2026Intelligent AI Delegationcited by 16
  16. 2021Training Value-Aligned Reinforcement Learning Agents Using a Normative Priorcited by 14
  17. 2019Inverse Reinforcement Learning in Contextual MDPscited by 12
  18. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safetycited by 12
  19. 2022Near-Optimal Multi-Agent Learning for Safe Coverage Controlcited by 11
  20. 2023Probabilistic Constraint for Safety-Critical Reinforcement Learningcited by 11
  21. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  22. 2021How RL Agents Behave When Their Actions Are Modifiedcited by 10
  23. 2023Human Control: Definitions and Algorithmscited by 10
  24. 2023Designing Fiduciary Artificial Intelligencecited by 8
  25. 2024Adaptive Language-Guided Abstraction from Contrastive Explanationscited by 8
  26. 2026Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive sciencecited by 8
  27. 2021Model-Free Risk-Sensitive Reinforcement Learningcited by 7
  28. 2023Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifestocited by 7
  29. 2021Understanding and Avoiding AI Failures: A Practical Guidecited by 6
  30. 2023Conceptual Framework for Autonomous Cognitive Entitiescited by 6
  31. 2025Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferencescited by 6
  32. 2021Axes for Sociotechnical Inquiry in AI Researchcited by 5
  33. 2021Incorporating Deception into CyberBattleSim for Autonomous Defensecited by 5
  34. 2022Categorical semantics of compositional reinforcement learningcited by 5
  35. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  36. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineeringcited by 5
  37. 2021Causal Analysis of Agent Behavior for AI Safetycited by 4
  38. 2022Policy Gradients for Probabilistic Constrained Reinforcement Learningcited by 4

50 most cited of 108 All 108 (JSON)