# CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs

arXiv: https://arxiv.org/abs/2606.11063 · HTML: https://mlsanity.com/paper/2606.11063/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding](https://mlsanity.com/paper/2009.03015/) (2020)
- [Large Language Models Often Know When They Are Being Evaluated](https://mlsanity.com/paper/2505.23836/) (2025)
- [Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement](https://mlsanity.com/paper/2402.11436/) (2024)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [Preventing Language Models From Hiding Their Reasoning](https://mlsanity.com/paper/2310.18512/) (2023)
- [Great Models Think Alike and this Undermines AI Oversight](https://mlsanity.com/paper/2502.04313/) (2025)
- [Ctrl-Z: Controlling AI Agents via Resampling](https://mlsanity.com/paper/2504.10374/) (2025)
- [Steering Evaluation-Aware Language Models to Act Like They Are Deployed](https://mlsanity.com/paper/2510.20487/) (2025)
- [Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols](https://mlsanity.com/paper/2409.07985/) (2024)
- [Reliable Weak-to-Strong Monitoring of LLM Agents](https://mlsanity.com/paper/2508.19461/) (2025)
- [The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness](https://mlsanity.com/paper/2505.14617/) (2025)
- [Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols](https://mlsanity.com/paper/2510.09462/) (2025)
- [PostMark: A Robust Blackbox Watermark for Large Language Models](https://mlsanity.com/paper/2406.14517/) (2024)
- [Self-Recognition in Language Models](https://mlsanity.com/paper/2407.06946/) (2024)
- [Evidence for Limited Metacognition in LLMs](https://mlsanity.com/paper/2509.21545/) (2025)
- [Evaluating Control Protocols for Untrusted AI Agents](https://mlsanity.com/paper/2511.02997/) (2025)
- [When can we trust untrusted monitoring? A safety case sketch across collusion strategies](https://mlsanity.com/paper/2602.20628/) (2026)
- [Attack Selection Reduces Safety in Concentrated AI Control Settings against Trusted Monitoring](https://mlsanity.com/paper/2602.04930/) (2026)
- [Can you Finetune your Binoculars? Embedding Text Watermarks into the Weights of Large Language Models](https://mlsanity.com/paper/2504.06446/) (2025)

## Built on by (most cited first, 3 of 3)

- [Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages](https://mlsanity.com/paper/2607.06596/) (2026)
- [Training LLMs to Verbalize Evaluation Awareness](https://mlsanity.com/paper/2609.36316/) (2026)
- [A Sense of Proprioception in Language Models](https://mlsanity.com/paper/2610.17495/) (2026)

## Abstract (arXiv)

AI control protocols oversee untrusted models by monitoring their actions and modifying potentially unsafe steps, often using a trusted model. This partially tampers with the untrusted model's trajectory. If the acting model detects such an intervention, it may infer properties of the monitor and adapt to evade the control protocol. We introduce CIAware-Bench, a benchmark for measuring control intervention (CI) awareness across frontier models. CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention. The benchmark comprises four task domains (essay writing, BigCodeBench, Bash Arena, and SHADE-Arena), with options to vary trajectory watermarking, side-task presence, and the control protocol. Evaluating thirteen acting models with eight intervening models shows substantial variation between settings and model pairs. CI awareness rises sharply for GPT-6 Astra and the Claude 5 models (Fable 5 and Opus 5). When explicitly probed, Astra reaches mean AUROC of 0.90 on Essay, 0.91 on BigCodeBench, 0.86 on Bash Arena and 0.77 on SHADE-Arena. Fable 5 and Opus 5 both reach 0.77 on Essay, with less consistent gains in the other environments. On SHADE-Arena, we demonstrate that telling an acting model that an action was replaced and asking it to adapt leads to moderate improvements in monitor evasion rates. In summary, our results highlight that control evaluations should already assume perfect CI awareness for conservative safety estimates, and that protocol design should explore countermeasures that make interventions harder to detect.