# A sketch of an AI control safety case

arXiv: https://arxiv.org/abs/2501.17315 · HTML: https://mlsanity.com/paper/2501.17315/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 23 of 23)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [Scheming AIs: Will AIs fake alignment during training in order to get power?](https://mlsanity.com/paper/2311.08379/) (2023)
- [Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs](https://mlsanity.com/paper/2407.04694/) (2024)
- [Preventing Language Models From Hiding Their Reasoning](https://mlsanity.com/paper/2310.18512/) (2023)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Safety Cases: How to Justify the Safety of Advanced AI Systems](https://mlsanity.com/paper/2403.10462/) (2024)
- [Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols](https://mlsanity.com/paper/2409.07985/) (2024)
- [Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats](https://mlsanity.com/paper/2411.17693/) (2024)
- [Safety case template for frontier AI: A cyber inability argument](https://mlsanity.com/paper/2411.08088/) (2024)
- [Risk thresholds for frontier AI](https://mlsanity.com/paper/2406.14713/) (2024)

## Built on by (most cited first, 30 of 39)

- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [Levels of Autonomy for AI Agents](https://mlsanity.com/paper/2506.12469/) (2025)
- [Evaluating Frontier Models for Stealth and Situational Awareness](https://mlsanity.com/paper/2505.01420/) (2025)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [How to evaluate control measures for LLM agents? A trajectory from today to superintelligence](https://mlsanity.com/paper/2504.05259/) (2025)
- [An alignment safety case sketch based on debate](https://mlsanity.com/paper/2505.03989/) (2025)
- [Control Tax: The Price of Keeping AI in Check](https://mlsanity.com/paper/2506.05296/) (2025)
- [Combining Cost-Constrained Runtime Monitors for AI Safety](https://mlsanity.com/paper/2507.15886/) (2025)
- [Do Large Language Models Know What They Are Capable Of?](https://mlsanity.com/paper/2512.24661/) (2025)
- [Measuring AI R&D Automation](https://mlsanity.com/paper/2603.03992/) (2026)
- [A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management](https://mlsanity.com/paper/2502.06656/) (2025)
- [Practical challenges of control monitoring in frontier AI deployments](https://mlsanity.com/paper/2512.22154/) (2025)
- [Systematic Hazard Analysis for Frontier AI using STPA](https://mlsanity.com/paper/2506.01782/) (2025)
- [Training Agents to Self-Report Misbehavior](https://mlsanity.com/paper/2602.22303/) (2026)
- [Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework](https://mlsanity.com/paper/2507.12872/) (2025)
- [Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors](https://mlsanity.com/paper/2607.07368/) (2026)
- [An Example Safety Case for Safeguards Against Misuse](https://mlsanity.com/paper/2505.18003/) (2025)
- [Removing Sandbagging in LLMs by Training with Weak Supervision](https://mlsanity.com/paper/2604.22082/) (2026)
- [MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring](https://mlsanity.com/paper/2605.09684/) (2026)
- [Training Deliberative Monitors for Black-Box Scheming Detection](https://mlsanity.com/paper/2605.29601/) (2026)
- ["Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms](https://mlsanity.com/paper/2606.12618/) (2026)
- [Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)](https://mlsanity.com/paper/2506.17846/) (2025)
- [How Transparent is DiffusionGemma?](https://mlsanity.com/paper/2606.20560/) (2026)
- [A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring](https://mlsanity.com/paper/2510.19476/) (2025)
- [Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands](https://mlsanity.com/paper/2605.15164/) (2026)
- [Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspective](https://mlsanity.com/paper/2602.07259/) (2026)
- [Clear, Compelling Arguments: Rethinking the Foundations of Frontier AI Safety Cases](https://mlsanity.com/paper/2603.08760/) (2026)
- [ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Data](https://mlsanity.com/paper/2604.17663/) (2026)
- [Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety](https://mlsanity.com/paper/2606.06529/) (2026)
- [Bootstrapped Monitoring: Leveraging Transparent Reasoning to Oversee Stronger AI Agents](https://mlsanity.com/paper/2606.11998/) (2026)

## Abstract (arXiv)

As LLM agents gain a greater capacity to cause harm, AI developers might increasingly rely on control measures such as monitoring to justify that they are safe. We sketch how developers could construct a "control safety case", which is a structured argument that models are incapable of subverting control measures in order to cause unacceptable outcomes. As a case study, we sketch an argument that a hypothetical LLM agent deployed internally at an AI company won't exfiltrate sensitive information. The sketch relies on evidence from a "control evaluation,"' where a red team deliberately designs models to exfiltrate data in a proxy for the deployment environment. The safety case then hinges on several claims: (1) the red team adequately elicits model capabilities to exfiltrate data, (2) control measures remain at least as effective in deployment, and (3) developers conservatively extrapolate model performance to predict the probability of data exfiltration in deployment. This safety case sketch is a step toward more concrete arguments that can be used to show that a dangerously capable LLM agent is safe to deploy.