# Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode

arXiv: https://arxiv.org/abs/2604.04978 · HTML: https://mlsanity.com/paper/2604.04978/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 4 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 7 of 7)

- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025)
- [WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks](https://mlsanity.com/paper/2504.18575/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [SafeToolBench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLMs](https://mlsanity.com/paper/2509.07315/) (2025)
- [Preemptive Detection and Correction of Misaligned Actions in LLM Agents](https://mlsanity.com/paper/2407.11843/) (2024)
- [AEGIS: No Tool Call Left Unchecked -- A Pre-Execution Firewall and Audit Layer for AI Agents](https://mlsanity.com/paper/2603.12621/) (2026)

## Built on by (most cited first, 10 of 10)

- [Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks](https://mlsanity.com/paper/2605.18583/) (2026)
- [Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development](https://mlsanity.com/paper/2605.20456/) (2026)
- [CAGE: Certified Authorization under Typed-Return Uncertainty for Tool-Using Agents](https://mlsanity.com/paper/2607.29190/) (2026)
- [SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries](https://mlsanity.com/paper/2608.12654/) (2026)
- [AI Snitches Get Glitches: Towards Evading Agentic Surveillance](https://mlsanity.com/paper/2606.25836/) (2026)
- [Task-Conditioned Least-Privilege Learning for Executable Terminal and MCP Agents](https://mlsanity.com/paper/2608.18351/) (2026)
- [Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?](https://mlsanity.com/paper/2608.27443/) (2026)
- [Approval Laundering: Systematizing Approval--Execution Binding Failures in AI Coding-Agent Harnesses](https://mlsanity.com/paper/2609.38983/) (2026)
- [Pincer: Resource Authorization for Agents using a Digital Twin](https://mlsanity.com/paper/2610.02569/) (2026)
- [AuthGateBench: Benchmarking LLMs as Authorization Gate](https://mlsanity.com/paper/2610.16426/) (2026)

## Abstract (arXiv)

Claude Code's auto mode is the first deployed permission system for AI coding agents, using a two-stage transcript classifier to gate dangerous tool calls. Anthropic reports a 0.4% false positive rate and 17% false negative rate on production traffic. We present the first independent evaluation of this system on deliberately ambiguous authorization scenarios, i.e., tasks where the user's intent is clear but the target scope, blast radius, or risk level is underspecified. Using AmPermBench, a 128-prompt benchmark spanning four DevOps task families and three controlled ambiguity axes, we evaluate 253 state-changing actions at the individual action level against oracle ground truth. Our findings characterize auto mode's scope-escalation coverage under this stress-test workload. The end-to-end false negative rate is 81.0% (95% CI: 73.8%-87.4%), substantially higher than the 17% reported on production traffic, reflecting a fundamentally different workload rather than a contradiction. Notably, 36.8% of all state-changing actions fall outside the classifier's scope via Tier 2 (in-project file edits), contributing to the elevated end-to-end FNR. Even restricting to the 160 actions the classifier actually evaluates (Tier 3), the FNR remains 70.3%, while the FPR rises to 31.9%. The Tier 2 coverage gap is most pronounced on artifact cleanup (92.9% FNR), where agents naturally fall back to editing state files when the expected CLI is unavailable. These results highlight a coverage boundary worth examining: auto mode assumes dangerous actions transit the shell, but agents routinely achieve equivalent effects through file edits that the classifier does not evaluate.