mlsanityData to 14 Oct 2026

A sketch of an AI control safety case

arXiv 2501.17315 ↗First posted 28 Jan 2025cs.AI cs.CR cs.SEcited by 39

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

As LLM agents gain a greater capacity to cause harm, AI developers might increasingly rely on control measures such as monitoring to justify that they are safe. We sketch how developers could construct a "control safety case", which is a structured argument that models are incapable of subverting control measures in order to cause unacceptable outcomes. As a case study, we sketch an argument that a hypothetical LLM agent deployed internally at an AI company won't exfiltrate sensitive information. The sketch relies on evidence from a "control evaluation,"' where a red team deliberately designs models to exfiltrate data in a proxy for the deployment environment. The safety case then hinges on several claims: (1) the red team adequately elicits model capabilities to exfiltrate data, (2) control measures remain at least as effective in deployment, and (3) developers conservatively extrapolate model performance to predict the probability of data exfiltration in deployment. This safety case sketch is a step toward more concrete arguments that can be used to show that a dangerously capable LLM agent is safe to deploy.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 23 of its 57 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

23 of 57 references.

References · 23

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  6. 2018AI safety via debatecited by 400
  7. 2024Frontier Models are Capable of In-context Schemingcited by 287
  8. 2023Pretraining Language Models with Human Preferencescited by 247
  9. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  10. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  11. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  12. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
11 more
  1. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  2. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  3. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  4. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  5. 2023Preventing Language Models From Hiding Their Reasoningcited by 65
  6. 2024Sabotage Evaluations for Frontier Modelscited by 52
  7. 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
  8. 2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocolscited by 31
  9. 2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 31
  10. 2024Safety case template for frontier AI: A cyber inability argumentcited by 19
  11. 2024Risk thresholds for frontier AIcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 20.

39 papers.

Sort:

By area:

  1. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 54
  2. 2025Levels of Autonomy for AI Agentscited by 48
  3. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  4. 2025Async Control: Stress-testing Asynchronous Control Measures for LLM Agentscited by 22
  5. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  6. 2025An alignment safety case sketch based on debatecited by 16
  7. 2025Control Tax: The Price of Keeping AI in Checkcited by 10
  8. 2025Combining Cost-Constrained Runtime Monitors for AI Safetycited by 10
  9. 2025Do Large Language Models Know What They Are Capable Of?cited by 10
  10. 2026Measuring AI R&D Automationcited by 9
  11. 2025A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Managementcited by 8
  12. 2025Practical challenges of control monitoring in frontier AI deploymentscited by 8
27 more
  1. 2025Systematic Hazard Analysis for Frontier AI using STPAcited by 6
  2. 2026Training Agents to Self-Report Misbehaviorcited by 6
  3. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 5
  4. 2026Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitorscited by 5
  5. 2025An Example Safety Case for Safeguards Against Misusecited by 4
  6. 2026Removing Sandbagging in LLMs by Training with Weak Supervisioncited by 4
  7. 2026MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoringcited by 4
  8. 2026Training Deliberative Monitors for Black-Box Scheming Detectioncited by 4
  9. 2026"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organismscited by 4
  10. 2025Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)cited by 3
  11. 2026How Transparent is DiffusionGemma?cited by 3
  12. 2025A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoringcited by 2
  13. 2026Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demandscited by 2
  14. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 1
  15. 2026Clear, Compelling Arguments: Rethinking the Foundations of Frontier AI Safety Casescited by 1
  16. 2026ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Datacited by 1
  17. 2026Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safetycited by 1
  18. 2026Bootstrapped Monitoring: Leveraging Transparent Reasoning to Oversee Stronger AI Agentscited by 1
  19. 2026More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversightcited by 1
  20. 2025Towards Measurement Theory for Artificial Intelligencecited by 0
  21. 2025Out-of-Context Abduction: LLMs Make Inferences About Procedural Data Leveraging Declarative Facts in Earlier Training Datacited by 0
  22. 2025Password-Activated Shutdown Protocols for Misaligned Frontier Agentscited by 0
  23. 2026What AI Red-Team Evaluations Can and Cannot Provecited by 0
  24. 2026Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Modelcited by 0
  25. 2026A Competing-Hazards Systematization of Loss of Control in Autonomous Agentscited by 0
  26. 2026Comprehension Audits to Mitigate Risks from Automated AI Researchcited by 0
  27. 2026AI Models Can Provably Hide Arbitrary Capabilitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agent Safety Should Be a Runtime Contract
  2. 2025Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Components
  3. 2026Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models
  4. 2024A Safe Harbor for AI Evaluation and Red Teaming
  5. 2026Towards Verifiably Safe Tool Use for LLM Agents
  6. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
  7. 2025Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods
  8. 2026FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs