mlsanityData to 14 Oct 2026

Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems

arXiv 2405.06624 ↗First posted 10 May 2024cs.AIcited by 96

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Ensuring that AI systems reliably and robustly avoid harmful or dangerous behaviours is a crucial challenge, especially for AI systems with a high degree of autonomy and general intelligence, or systems used in safety-critical contexts. In this paper, we will introduce and define a family of approaches to AI safety, which we will refer to as guaranteed safe (GS) AI. The core feature of these approaches is that they aim to produce AI systems which are equipped with high-assurance quantitative safety guarantees. This is achieved by the interplay of three core components: a world model (which provides a mathematical description of how the AI system affects the outside world), a safety specification (which is a mathematical description of what effects are acceptable), and a verifier (which provides an auditable proof certificate that the AI satisfies the safety specification relative to the world model). We outline a number of approaches for creating each of these three core components, describe the main technical challenges, and suggest a number of potential solutions to them. We also argue for the necessity of this approach to AI safety, and for the inadequacy of the main alternative approaches.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 73 of its 160 references that are in the map. First posted on arXiv 10 May 2024.

Builds on

73 of 160 references.

References · 73

  1. 2016Concrete Problems in AI Safetycited by 1,642
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  4. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 586
  5. 2018Pyro: Deep Universal Probabilistic Programmingcited by 380
  6. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  8. 2020A Survey on Neural Network Interpretabilitycited by 291
  9. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  10. 2022Autoformalization with Large Language Modelscited by 263
  11. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  12. 2023An Overview of Catastrophic AI Riskscited by 222
38 more
  1. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 165
  2. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  3. 2018Categorizing Variants of Goodhart's Lawcited by 149
  4. 2020Algorithmic Fairnesscited by 143
  5. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  6. 2016The Off-Switch Gamecited by 97
  7. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  8. 2018Scenic: A Language for Scenario Specification and Scene Generationcited by 83
  9. 2021Introduction to Neural Network Verificationcited by 63
  10. 2020Trustworthy AIcited by 62
  11. 2021Identifiability in inverse reinforcement learningcited by 55
  12. 2021Agent Incentives: A Causal Perspectivecited by 51
  13. 2020Is SGD a Bayesian sampler? Well, almostcited by 50
  14. 2023Joint Bayesian Inference of Graphical Structure and Parameters with a Single Generative Flow Networkcited by 50
  15. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 49
  16. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  17. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45
  18. 2019Liability, Ethics, and Culture-Aware Behavior Specification using Rulebookscited by 43
  19. 2021An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defencescited by 42
  20. 2018Learning Linear Temporal Propertiescited by 41
  21. 2020Scenic: A Language for Scenario Specification and Data Generationcited by 40
  22. 2020Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real Worldcited by 39
  23. 2020Understanding Learned Reward Functionscited by 37
  24. 2016Towards Verified Artificial Intelligencecited by 36
  25. 2019Certified Reinforcement Learning with Logic Guidancecited by 36
  26. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 36
  27. 2023Learning adaptive planning representations with natural language guidancecited by 36
  28. 2015Safety-Constrained Reinforcement Learning for MDPscited by 33
  29. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  30. 2017Compositional Falsification of Cyber-Physical Systems with Machine Learning Componentscited by 32
  31. 2022Reward Gaming in Conditional Text Generationcited by 30
  32. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
  33. 20213DP3: 3D Scene Perception via Probabilistic Programmingcited by 26
  34. 2019Neural networks are a priori biased towards Boolean functions with low entropycited by 24
  35. 2022A Hazard Analysis Framework for Code Synthesis Large Language Modelscited by 22
  36. 2020Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAIcited by 21
  37. 2024Opening the AI black box: program synthesis via mechanistic interpretabilitycited by 20
  38. 2019RowHammer: A Retrospectivecited by 19

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 35; 2026: 37.

96 papers.

Sort:

By area:

  1. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  4. 2024Large Language Model Safety: A Holistic Surveycited by 59
  5. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  6. 2025Infrastructure for AI Agentscited by 41
  7. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  8. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  9. 2024Agentic Information Retrievalcited by 21
  10. 2026Large Language Model Reasoning Failurescited by 21
  11. 2024LiSeCo: Linear Semantic Control for Language Generationcited by 20
  12. 2024Compact Proofs of Model Performance via Mechanistic Interpretabilitycited by 18
38 more
  1. 2025LLM Strategic Reasoning: Agentic Study through Behavioral Game Theorycited by 17
  2. 2025General agents contain world modelscited by 17
  3. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 14
  4. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  5. 2026When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agentscited by 14
  6. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 13
  7. 2025Learning Safety Constraints for Large Language Modelscited by 13
  8. 2024Imagining and building wise machines: The centrality of AI metacognitioncited by 12
  9. 2024SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behaviorcited by 10
  10. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
  11. 2025A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Managementcited by 8
  12. 2025Resource Rational Contractualism Should Guide AI Alignmentcited by 8
  13. 2025Guaranteed prediction sets for functional surrogate modelscited by 7
  14. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  15. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  16. 2025VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Codecited by 7
  17. 2026Safe Exploration via Policy Priorscited by 6
  18. 2025The Other Mind: How Language Models Exhibit Human Temporal Cognitioncited by 5
  19. 2026Breaking Up with Normatively Monolithic Agency with GRACE: A Reason-Based Neuro-Symbolic Architecture for Safe and Ethical AI Alignmentcited by 5
  20. 2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
  21. 2024Policy Verification in Stochastic Dynamical Systems Using Logarithmic Neural Certificatescited by 4
  22. 2024Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuitscited by 4
  23. 2024Securing the Future of GenAI: Policy and Technologycited by 4
  24. 2024Investigating Representation Universality: Case Study on Genealogical Representationscited by 4
  25. 2025Where AI Assurance Might Go Wrong: Initial lessons from engineering of critical systemscited by 4
  26. 2025On the creation of narrow AI: hierarchy and nonlocality of neural network skillscited by 4
  27. 2025From monoliths to modules: Decomposing transducers for efficient world modellingcited by 4
  28. 2026Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoningcited by 4
  29. 2026The Two Boundaries: Why Behavioral AI Governance Fails Structurallycited by 4
  30. 2026What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agentscited by 4
  31. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architecturescited by 3
  32. 2025AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretabilitycited by 3
  33. 2025The Limits of Predicting Agents from Behaviourcited by 3
  34. 2025Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)cited by 3
  35. 2025Limits of Safe AI Deployment: Differentiating Oversight and Controlcited by 3
  36. 2026Stable Agentic Control: Tool-Mediated LLM Architecture for Autonomous Cyber Defensecited by 3
  37. 2025Revisiting Rogers' Paradox in the Context of Human-AI Interactioncited by 2
  38. 2025Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Studycited by 2

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Domain-Agnostic Scalable AI Safety Ensuring Framework
  2. 2026SafePro: Evaluating the Safety of Professional-Level AI Agents
  3. 2025A Safety and Security Framework for Real-World Agentic Systems
  4. 2023Provably safe systems: the only path to controllable AGI
  5. 2024SafeEmbodAI: a Safety Framework for Mobile Robots in Embodied AI Systems
  6. 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
  7. 2025Security practices in AI development
  8. 2024AI Risk Management Should Incorporate Both Safety and Security