# The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior arXiv: https://arxiv.org/abs/2608.11386 · HTML: https://mlsanity.com/paper/2608.11386/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 36 of its 59 references that are in the map. - First posted on arXiv 11 Aug 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 30 of 36) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024) - [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020) - [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024) - [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024) - [MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers](https://mlsanity.com/paper/2508.20453/) (2025) - [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025) - [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025) - [RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph](https://mlsanity.com/paper/2410.14684/) (2024) - [SWE-bench Goes Live!](https://mlsanity.com/paper/2505.23419/) (2025) - [GTA: A Benchmark for General Tool Agents](https://mlsanity.com/paper/2407.08713/) (2024) - [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024) - [Towards a Science of AI Agent Reliability](https://mlsanity.com/paper/2602.16666/) (2026) - [On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub](https://mlsanity.com/paper/2509.14745/) (2025) - [The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason](https://mlsanity.com/paper/2506.12286/) (2025) - [Reducing Cost of LLM Agents with Trajectory Reduction](https://mlsanity.com/paper/2509.23586/) (2025) - [ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions](https://mlsanity.com/paper/2601.06112/) (2026) - [Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories](https://mlsanity.com/paper/2511.00197/) (2025) - [MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmark](https://mlsanity.com/paper/2508.07575/) (2025) - [ContextBench: A Benchmark for Context Retrieval in Coding Agents](https://mlsanity.com/paper/2602.05892/) (2026) - [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025) - [AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition](https://mlsanity.com/paper/2602.11348/) (2026) - [Improving Code Localization with Repository Memory](https://mlsanity.com/paper/2510.01003/) (2025) - [Agentic Refactoring: An Empirical Study of AI Coding Agents](https://mlsanity.com/paper/2511.04824/) (2025) - [On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents](https://mlsanity.com/paper/2601.20404/) (2026) - [Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study](https://mlsanity.com/paper/2506.08311/) (2025) - [An Empirical Study on Failures in Automated Issue Solving](https://mlsanity.com/paper/2509.13941/) (2025) ## Built on by (most cited first, 3 of 3) - [The Empire, Long Divided, Must Unite: Architectural Convergence in Three LLM Agent Harnesses](https://mlsanity.com/paper/2608.23953/) (2026) - [Growing an Agent/Prover Interface: Evolutionary Tool Design for Cost-Efficient Theorem Proving in Rocq and Lean](https://mlsanity.com/paper/2609.39544/) (2026) - [State Representation Shapes Long-Horizon Agent Learning and Transfer: An Empirical Study](https://mlsanity.com/paper/2610.15407/) (2026) ## Abstract (arXiv) As large language models continue to improve, agentic systems are becoming increasingly important, and tools are a key design dimension because they determine how agents access information and take action in their environments. Prior work on agent tooling has primarily focused on expanding what agents can do, but has paid less systematic attention to how those capabilities are organized and exposed to the model. We refer to this latter design dimension as tool architecture. We study tool architecture in coding agents through controlled experiments on repository-level issue fixing, comparing six tool architectures that hold the underlying information and actions similar while varying how they are organized and exposed to the model, across three actors and a total of 11,700 trajectories. Our experiments show that, even when tools provide similar capabilities, tool architecture changes agent behavior: Compared to a basic architecture where the agent has only the bash tool, more structured low-level interfaces improve consistency across repeated attempts by up to 4.7 ×; natural-language search broadens repository exploration and increases access to relevant files by more than 11%; and Python CodeAct-style interfaces achieve similar task performance with 41.6% fewer steps and 56.3% lower token usage. By contrast, lightweight text-based cognitive-scaffolding tools, such as tools that let the agent record intermediate reasoning, have limited effect on actor behavior.