mlsanityData to 14 Oct 2026

AI Control: Improving Safety Despite Intentional Subversion

arXiv 2312.06942 ↗First posted 12 Dec 2023cs.LGcited by 229

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

As large language models (LLMs) become more powerful and are deployed more autonomously, it will be increasingly important to prevent them from causing harmful outcomes. Researchers have investigated a variety of safety techniques for this purpose, e.g. using models to review the outputs of other models, or red-teaming techniques to surface subtle failure modes. However, researchers have not evaluated whether such techniques still ensure safety if the model is itself intentionally trying to subvert them. In this paper, we develop and evaluate pipelines of safety techniques ("protocols") that are robust to intentional subversion. We investigate a scenario in which we want to solve a sequence of programming problems, using access to a powerful but untrusted model (in our case, GPT-4), access to a less powerful trusted model (in our case, GPT-3.5), and limited access to high-quality trusted labor. We investigate protocols that aim to never submit solutions containing backdoors, which we operationalize here as logical errors that are not caught by test cases. We investigate a range of protocols and test each against strategies that the untrusted model could use to subvert them. One protocol is what we call trusted editing. This protocol first asks GPT-4 to write code, and then asks GPT-3.5 to rate the suspiciousness of that code. If the code is below some suspiciousness threshold, it is submitted. Otherwise, GPT-3.5 edits the solution to remove parts that seem suspicious and then submits the edited code. Another protocol is untrusted monitoring. This protocol asks GPT-4 to write code, and then asks another instance of GPT-4 whether the code is backdoored, using various techniques to prevent the GPT-4 instances from colluding. These protocols improve substantially on simple baselines.

Key facts. Cited by 229 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 22 of its 58 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

22 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2021Measuring Coding Challenge Competence With APPScited by 902
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2023A Watermark for Large Language Modelscited by 725
  9. 2018AI safety via debatecited by 400
  10. 2022Self-critiquing models for assisting human evaluatorscited by 380
  11. 2023AI Alignment: A Comprehensive Surveycited by 263
  12. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
9 more
  1. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  2. 2023On the Reliability of Watermarks for Large Language Modelscited by 189
  3. 2023Model evaluation for extreme riskscited by 177
  4. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  5. 2023Preventing Language Models From Hiding Their Reasoningcited by 65
  6. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  7. 2023Testing Language Model Agents Safely in the Wildcited by 46
  8. 2023Debate Helps Supervise Unreliable Expertscited by 43
  9. 2019A Research Agenda: Dynamic Models to Defend Against Correlated Attackscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 71; 2026: 139.

229 papers.

Sort:

By area:

  1. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  2. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  3. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
  4. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  5. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
  6. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  7. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 88
  8. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  9. 2024Visibility into AI Agentscited by 79
  10. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 76
  11. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 71
  12. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 71
38 more
  1. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 55
  2. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 54
  3. 2025Detecting High-Stakes Interactions with Activation Probescited by 53
  4. 2024Sabotage Evaluations for Frontier Modelscited by 52
  5. 2026Building Production-Ready Probes For Geminicited by 47
  6. 2025Ctrl-Z: Controlling AI Agents via Resamplingcited by 44
  7. 2025Infrastructure for AI Agentscited by 41
  8. 2025A sketch of an AI control safety casecited by 39
  9. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  10. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  11. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  12. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  13. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 32
  14. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 32
  15. 2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocolscited by 31
  16. 2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 31
  17. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  18. 2025Among Us: A Sandbox for Measuring and Detecting Agentic Deceptioncited by 26
  19. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 25
  20. 2025Scaling Laws For Scalable Oversightcited by 23
  21. 2025Async Control: Stress-testing Asynchronous Control Measures for LLM Agentscited by 22
  22. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  23. 2025A benchmark for vericoding: formally verified program synthesiscited by 20
  24. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 20
  25. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  26. 2025COSMIC: Generalized Refusal Direction Identification in LLM Activationscited by 19
  27. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 19
  28. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 18
  29. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbositycited by 17
  30. 2025Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalizationcited by 16
  31. 2024Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?cited by 15
  32. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 15
  33. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 15
  34. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  35. 2026Detecting Safety Violations Across Many Agent Tracescited by 14
  36. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internalscited by 13
  37. 2025Learning Safety Constraints for Large Language Modelscited by 13
  38. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 12

50 most cited of 229 All 229 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering
  2. 2024Large Language Model Safety: A Holistic Survey
  3. 2024Safeguarding AI Agents: Developing and Analyzing Safety Architectures
  4. 2025Security practices in AI development
  5. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing
  6. 2026What Matters For Safety Alignment?
  7. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  8. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models