# MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring

arXiv: https://arxiv.org/abs/2605.09684 · HTML: https://mlsanity.com/paper/2605.09684/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 10 May 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 8 of 8)

- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [Ctrl-Z: Controlling AI Agents via Resampling](https://mlsanity.com/paper/2504.10374/) (2025)
- [A sketch of an AI control safety case](https://mlsanity.com/paper/2501.17315/) (2025)
- [Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols](https://mlsanity.com/paper/2510.09462/) (2025)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [BashArena: A Control Setting for Highly Privileged AI Agents](https://mlsanity.com/paper/2512.15688/) (2025)
- [LinuxArena: A Control Setting for AI Agents in Live Production Software Environments](https://mlsanity.com/paper/2604.15384/) (2026)
- [Optimizing AI Agent Attacks With Synthetic Data](https://mlsanity.com/paper/2511.02823/) (2025)

## Built on by (most cited first, 4 of 4)

- [SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors](https://mlsanity.com/paper/2605.16626/) (2026)
- [RH-Detect: A Unified Benchmark for Reward Hacking Detection](https://mlsanity.com/paper/2610.10947/) (2026)
- [Code Monitor Red Teaming for Public-Test-Passing Code](https://mlsanity.com/paper/2607.20852/) (2026)
- [Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents](https://mlsanity.com/paper/2609.19587/) (2026)

## Abstract (arXiv)

We introduce a red-teaming methodology that exposes harder-to-catch attacks for coding-agent monitors, suggesting that current practices may under-elicit attacks and overstate monitor performance. We identify three challenges with current red-teaming. First, mode collapse in attack generation, which we reduce with a novel attack taxonomy for broader coverage. Second, a conceive-execute gap: frontier LLMs can propose strong attack ideas or execute them, but not all at once. We mitigate this by decomposing attack construction into strategy generation, execution, and post-hoc trajectory refinement. Third, manual elicitation is costly to scale, which we address with our semi-automated red-teaming pipeline. Applied to BashArena, an AI control setting for tool-using coding agents, this pipeline produces MonitoringBench, a benchmark of 2,644 attack trajectories for evaluating monitor capabilities and failure modes. Our pipeline produces more diverse and stronger attacks: Opus-4.5 monitor's catch rate falls from 94.9% on elicited-only Opus attacks to 60.3% on our best refined attacks, with larger drops for several mid-tier monitors. Attacks optimized against three development monitors generalize to ten held-out monitors, with catch rates generally increasing with monitor capability. Using this benchmark, we provide a snapshot of the current monitor capabilities and find that frontier monitors often detect suspicious actions but fall for persuasion or fail to calibrate suspiciousness scores appropriately, suggesting tractable paths for improvement. MonitoringBench provides both a static benchmark for current tool-use monitors and a reusable methodology for refreshing these evaluations as agents and monitors improve.