# Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?

arXiv: https://arxiv.org/abs/2412.12480 · HTML: https://mlsanity.com/paper/2412.12480/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 10 of 10)

- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents](https://mlsanity.com/paper/2406.06613/) (2024)
- [Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs](https://mlsanity.com/paper/2407.04694/) (2024)
- [Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols](https://mlsanity.com/paper/2409.07985/) (2024)
- [Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats](https://mlsanity.com/paper/2411.17693/) (2024)
- [Estimating the Probabilities of Rare Outputs in Language Models](https://mlsanity.com/paper/2410.13211/) (2024)

## Built on by (most cited first, 15 of 15)

- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [Evaluating Frontier Models for Stealth and Situational Awareness](https://mlsanity.com/paper/2505.01420/) (2025)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [How to evaluate control measures for LLM agents? A trajectory from today to superintelligence](https://mlsanity.com/paper/2504.05259/) (2025)
- [Control Tax: The Price of Keeping AI in Check](https://mlsanity.com/paper/2506.05296/) (2025)
- [Practical challenges of control monitoring in frontier AI deployments](https://mlsanity.com/paper/2512.22154/) (2025)
- [Frontier Models Can Take Actions at Low Probabilities](https://mlsanity.com/paper/2603.02202/) (2026)
- [Distributed Attacks in Persistent-State AI Control](https://mlsanity.com/paper/2607.02514/) (2026)
- [Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors](https://mlsanity.com/paper/2607.07368/) (2026)
- [Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)](https://mlsanity.com/paper/2506.17846/) (2025)
- [Subversion via Focal Points: Investigating Collusion in LLM Monitoring](https://mlsanity.com/paper/2507.03010/) (2025)
- [Basic Legibility Protocols Improve Trusted Monitoring](https://mlsanity.com/paper/2602.10153/) (2026)
- [Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety](https://mlsanity.com/paper/2606.06529/) (2026)
- [Bootstrapped Monitoring: Leveraging Transparent Reasoning to Oversee Stronger AI Agents](https://mlsanity.com/paper/2606.11998/) (2026)
- [Shared and structured inputs undermine collective random choice by reasoning AI agents](https://mlsanity.com/paper/2610.09667/) (2026)

## Abstract (arXiv)

An AI control protocol is a plan for usefully deploying AI systems that aims to prevent an AI from intentionally causing some unacceptable outcome. This paper investigates how well AI systems can generate and act on their own strategies for subverting control protocols whilst operating statelessly (without shared memory between contexts). To do this, an AI system may need to reliably generate optimal plans in each context, take actions with well-calibrated probabilities, and coordinate plans with other instances of itself without communicating. We develop Subversion Strategy Eval, a suite of eight environments, covering a range of protocols and strategic capabilities, and six sets of affordances that help isolate individual capabilities. We implement the evaluation in Inspect-AI and release it open-source. We evaluate Claude 3.5 models, including helpful-only versions, as well as OpenAI reasoning models. None of the models demonstrate substantial capability in strategizing to subvert control protocols statelessly. However, providing models with additional affordances, such as the ability to share a plan between contexts, can substantially improve performance. We hope our evaluations can act as a leading indicator for when models are capable of subverting control protocols and also relax the worst-case assumption of perfect strategic ability in AI control evaluations.