mlsanityData to 7 Oct 2026

Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks

arXiv 2605.18583 ↗First posted 18 May 2026cs.SE cs.AI cs.CLcited by 8

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Coding agents now run autonomously with shell, file, and network privileges. When a user issues a benign request, the agent sometimes does more than asked: it deletes unrelated files, wipes a stale credentials backup, or rewrites configuration the user never mentioned. We call these scope expansions overeager actions, an authorization problem distinct from capability failures, prompt injection, or sandbox escapes. We present OverEager-Gen, a benchmark dedicated to overeager behavior on benign tasks. Building it surfaces a measurement-validity issue: if a benchmark spells out the authorized scope inside the prompt, the agent stops inferring boundaries and starts pattern-matching declaration text. On Claude Code, stripping the consent declaration alone raises the overeager rate from 0.0% to 17.1% on paired scenarios (McNemar exact p = 2.4 x 10^-4). OverEager-Gen therefore certifies each scenario's discriminative power before admission via a behavioral-gradient validator, audits internal tool calls through a dual-channel stack (PATH-injected shim plus per-agent event streams), and ships byte-identical consent_kept and consent_stripped variants. OverEager-Bench contains 500 validated scenarios and ~7,500 runs across four agent products (Claude Code, OpenHands, Codex CLI, Gemini CLI) and six base models; a 50-sample re-annotation gives Cohen's kappa = 0.73 and rule-judge recall = 1.00. Stripping consent multiplies the overeager rate on every shared base model (Delta in [11.9, 17.2] pp). The framework axis dominates effect size: a permissive cluster (Claude Code, Codex CLI, Gemini CLI) runs at 5.4-27.7% while the ask-to-continue framework (OpenHands) sits at 0.2-4.5% (Fisher p <= 10^-5). Within-framework base-model variance reaches 15.9 pp, indicating that model-layer alignment does not fully propagate through permissive permission gating.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 15 of its 32 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

15 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode4 Apr
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  4. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  5. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 10

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  7. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  8. 2025PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literaturecited by 15
  9. 2026ClawEnvKit: Automatic Environment Generation for Claw-Like Agentscited by 6
  10. 2026False Friends in the Shell: Unveiling the Emoticon Semantic Confusion in Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantifying Overclaiming Propensity in Frontier LLM Agents
  2. 2026SkillScope: Toward Fine-Grained Least-Privilege Enforcement for Agent Skills
  3. 2026SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors
  4. 2026Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions
  5. 2026AgentRaft: Automated Detection of Data Over-Exposure in LLM Agents
  6. 2026ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
  7. 2026Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents
  8. 2026ScopeBench: Do Agents Preserve Engagement Boundaries Under Goal Pressure?