mlsanityData to 7 Oct 2026

Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols

arXiv 2409.07985 ↗First posted 12 Sep 2024cs.AI cs.LGcited by 28

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

To evaluate the safety and usefulness of deployment protocols for untrusted AIs, AI Control uses a red-teaming exercise played between a protocol designer and an adversary. This paper introduces AI-Control Games, a formal decision-making model of the red-teaming exercise as a multi-objective, partially observable, stochastic game. We also introduce reductions from AI-Control Games to a special case of zero-sum partially observable stochastic games that allow us to leverage existing algorithms to find Pareto-optimal protocols. We apply our formalism to model, evaluate and synthesise protocols for deploying untrusted language models as programming assistants, focusing on Trusted Monitoring protocols, which use weaker language models and limited human assistance. To demonstrate the utility of our formalism, we show improvements over empirical studies in existing settings, evaluate protocols in new settings, and analyse how modelling assumptions affect the safety and usefulness of protocols. Finally, we leverage our formalism to precisely describe some of the implicit assumptions in prior control work.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 9 of its 48 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

9 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Training8 Mar
  2. 2023AI Control: Improving Safety Despite Intentional Subversion12 Dec

Other references · 7

  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  3. 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
  4. 2020Solving Zero-Sum One-Sided Partially Observable Stochastic Gamescited by 12
  5. 2022HSVI can solve zero-sum Partially Observable Stochastic Gamescited by 8
  6. 2020Double Oracle Algorithm for Computing Equilibria in Continuous Gamescited by 7
  7. 2021The Stackelberg Equilibrium for One-sided Zero-sum Partially Observable Stochastic Gamescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 14.

28 papers.

Sort:

By area:

  1. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  2. 2025Ctrl-Z: Controlling AI Agents via Resamplingcited by 40
  3. 2025A sketch of an AI control safety casecited by 36
  4. 2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 27
  5. 2025Async Control: Stress-testing Asynchronous Control Measures for LLM Agentscited by 18
  6. 2024Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?cited by 14
  7. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  8. 2025Evaluating Control Protocols for Untrusted AI Agentscited by 9
  9. 2025Control Tax: The Price of Keeping AI in Checkcited by 7
  10. 2025Optimizing AI Agent Attacks With Synthetic Datacited by 6
  11. 2025Practical challenges of control monitoring in frontier AI deploymentscited by 6
  12. 2026Attack Selection Reduces Safety in Concentrated AI Control Settings against Trusted Monitoringcited by 6
16 more
  1. 2026LinuxArena: A Control Setting for AI Agents in Live Production Software Environmentscited by 6
  2. 2025Scheming Ability in LLM-to-LLM Strategic Interactionscited by 4
  3. 2026Distributed Attacks in Persistent-State AI Controlcited by 3
  4. 2026AI Alignment via Incentives and Correctioncited by 1
  5. 2026Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safetycited by 1
  6. 2026CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMscited by 1
  7. 2026What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agentscited by 1
  8. 2025Password-Activated Shutdown Protocols for Misaligned Frontier Agentscited by 0
  9. 2025The Double Life of Code World Models: Provably Unmasking Malicious Behavior Through Execution Tracescited by 0
  10. 2026DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigationcited by 0
  11. 2026Democratizing Agent Deployment Safety: A Structural Monitoring Approachcited by 0
  12. 2026Sound Probabilistic Safety Bounds for Large Language Modelscited by 0
  13. 2026Code Monitor Red Teaming for Public-Test-Passing Codecited by 0
  14. 2026Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Modelcited by 0
  15. 2026Improving scalable oversight with co-trained monitorscited by 0
  16. 2026When Does Randomized Oversight Align AI Agents That Can Conceal?cited by 0