SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents
On the map: LLM agents and tool use → Software engineering coding agents
also in 1 more
Abstract (arXiv)
A coding agent executes a benign task as a sequence of shell, file, and network actions, any of which can quietly exceed the authorized scope while the task still completes. We call this overeager behavior: the prompt is not adversarial and the run succeeds, yet an out-of-scope step can leak credentials or delete files. Existing benchmarks miss it: task-completion suites credit any finished run, jailbreak suites probe adversarial prompts, and the one prior overeager benchmark applies a single fixed prompt set to every agent-model pair, leaving its easiest and most resistant pairs under-measured. We present SNARE (Synthesizing Non-adversarial scenarios for Adaptive Reward-guided Elicitation), a pipeline that composes benign scenarios from reusable scope and trap fragments, scores each run with a judge-free oracle flagging trap-pattern matches and unsolicited file additions or deletions, and uses Thompson sampling to steer each pair's run budget toward the scenarios that most often trigger it. Instantiating it over 24 overeager archetypes yields OverEager, which we run across a 4x5 matrix of four coding agents and five base models. Across 10,000 benign runs, 19.51% trigger overeager behavior, with per-pair rates spanning 11.9x. This variation is driven by the agent framework, not the model: the framework accounts for 56% of it against the model's 21%, so any single-framework or single-model evaluation undercounts the matrix by about a fifth.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 35 references that are in the map. First posted on arXiv 27 May 2026.
Builds on
13 of 35 references.
Closest ancestors · 1
Same direction, newest first
Other references · 12
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2014Learning to Optimize via Information-Directed Samplingcited by 168
- 2026Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Taskscited by 13
- 2026An Empirical Study of Proactive Coding Assistants in Real-World Software Developmentcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026MemoHarness: Agent Harnesses That Learn from Experience
- 2026Coding Agents are Strong Prompt Optimizers
- 2026TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking
- 2026EASE: Behavior-Adaptive Skill Curation for Self-Evolving Agents
- 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
- 2026Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams
- 2026SkillZip: Evaluation-Free Skill Compression for Self-Evolving Agents by Discovering Reusable Structure
- 2026Adapting Context Compression for Long-Horizon Agents with Counterfactual Continuations