When Does Restricting a Coding Agent to executecode Help? A Regime × Agent-Design Ablation
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 46 references that are in the map. First posted on arXiv 12 Jul 2026.
Builds on
17 of 46 references.
Closest ancestors · 8
Same direction, newest first
- 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution8 May
- 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses28 Apr
- 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems14 Apr
- 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures3 Apr
- 2025Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases11 Dec
- 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?17 Nov
- 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents11 Apr
- 2024Executable Code Actions Elicit Better LLM Agents1 Feb
Other references · 9
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
- 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
- 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
- 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
- 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 44
- 2025ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filteringcited by 19
- 2026ToolTree: Efficient LLM Agent Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional Pruningcited by 19
- 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 17
- 2026Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agentscited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026What Does a Harness Buy? Tokens, Mostlycited by 0
- 2026What Does a Harness Buy? Tokens, Mostly3 Oct 2026
Similar, not cited
Close in wording, no citation either way
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities
- 2026The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Task
- 2026Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents
- 2026Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
- 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
- 2026AgentSWE: Can Coding Agents Build the Agent You Actually Want?
- 2026Code Isn't Memory: A Structural Codebase Index Inside a Coding Agent
- 2026Coding Agents Don't Know When to Act