# AI Control: Improving Safety Despite Intentional Subversion

arXiv: https://arxiv.org/abs/2312.06942 · HTML: https://mlsanity.com/paper/2312.06942/

## Key facts

- Cited by 229 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 22 of its 58 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [On the Reliability of Watermarks for Large Language Models](https://mlsanity.com/paper/2306.04634/) (2023)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Scheming AIs: Will AIs fake alignment during training in order to get power?](https://mlsanity.com/paper/2311.08379/) (2023)
- [Preventing Language Models From Hiding Their Reasoning](https://mlsanity.com/paper/2310.18512/) (2023)
- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [Testing Language Model Agents Safely in the Wild](https://mlsanity.com/paper/2311.10538/) (2023)
- [Debate Helps Supervise Unreliable Experts](https://mlsanity.com/paper/2311.08702/) (2023)
- [A Research Agenda: Dynamic Models to Defend Against Correlated Attacks](https://mlsanity.com/paper/1903.06293/) (2019)

## Built on by (most cited first, 30 of 229)

- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [Weak-to-Strong Jailbreaking on Large Language Models](https://mlsanity.com/paper/2401.17256/) (2024)
- [When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors](https://mlsanity.com/paper/2507.05246/) (2025)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [MALT: Improving Reasoning with Multi-Agent LLM Training](https://mlsanity.com/paper/2412.01928/) (2024)
- [Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents](https://mlsanity.com/paper/2505.02077/) (2025)
- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs](https://mlsanity.com/paper/2410.03768/) (2024)
- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [Detecting High-Stakes Interactions with Activation Probes](https://mlsanity.com/paper/2506.10805/) (2025)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Building Production-Ready Probes For Gemini](https://mlsanity.com/paper/2601.11516/) (2026)
- [Ctrl-Z: Controlling AI Agents via Resampling](https://mlsanity.com/paper/2504.10374/) (2025)
- [Infrastructure for AI Agents](https://mlsanity.com/paper/2501.10114/) (2025)
- [A sketch of an AI control safety case](https://mlsanity.com/paper/2501.17315/) (2025)
- [Evaluating Frontier Models for Stealth and Situational Awareness](https://mlsanity.com/paper/2505.01420/) (2025)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Evaluating and Steering Modality Preferences in Multimodal Large Language Model](https://mlsanity.com/paper/2505.20977/) (2025)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [Large language models can learn and generalize steganographic chain-of-thought under process supervision](https://mlsanity.com/paper/2506.01926/) (2025)
- [Adversarial Attacks on Robotic Vision Language Action Models](https://mlsanity.com/paper/2506.03350/) (2025)
- [Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols](https://mlsanity.com/paper/2409.07985/) (2024)
- [Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats](https://mlsanity.com/paper/2411.17693/) (2024)
- [Reliable Weak-to-Strong Monitoring of LLM Agents](https://mlsanity.com/paper/2508.19461/) (2025)
- [Among Us: A Sandbox for Measuring and Detecting Agentic Deception](https://mlsanity.com/paper/2504.04072/) (2025)

## Abstract (arXiv)

As large language models (LLMs) become more powerful and are deployed more autonomously, it will be increasingly important to prevent them from causing harmful outcomes. Researchers have investigated a variety of safety techniques for this purpose, e.g. using models to review the outputs of other models, or red-teaming techniques to surface subtle failure modes. However, researchers have not evaluated whether such techniques still ensure safety if the model is itself intentionally trying to subvert them. In this paper, we develop and evaluate pipelines of safety techniques ("protocols") that are robust to intentional subversion. We investigate a scenario in which we want to solve a sequence of programming problems, using access to a powerful but untrusted model (in our case, GPT-4), access to a less powerful trusted model (in our case, GPT-3.5), and limited access to high-quality trusted labor. We investigate protocols that aim to never submit solutions containing backdoors, which we operationalize here as logical errors that are not caught by test cases. We investigate a range of protocols and test each against strategies that the untrusted model could use to subvert them. One protocol is what we call trusted editing. This protocol first asks GPT-4 to write code, and then asks GPT-3.5 to rate the suspiciousness of that code. If the code is below some suspiciousness threshold, it is submitted. Otherwise, GPT-3.5 edits the solution to remove parts that seem suspicious and then submits the edited code. Another protocol is untrusted monitoring. This protocol asks GPT-4 to write code, and then asks another instance of GPT-4 whether the code is backdoored, using various techniques to prevent the GPT-4 instances from colluding. These protocols improve substantially on simple baselines.