What Does a Harness Buy? Tokens, Mostly
New in LLM agents and tool use; no direction yet. LLM agents and tool use
Abstract (arXiv)
A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. We run five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the preamble of system prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider's price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 3 Oct 2026.
Builds on
20 of 26 references.
References · 20
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
- 2024AI Agents That Mattercited by 219
- 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
- 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflowscited by 57
- 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 34
- 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architecturescited by 18
- 2026The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluationcited by 12
- 2026Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Taskscited by 8
- 2026An Empirical Study of Harness Design for Coding Agentscited by 6
8 more
- 2026Scaffold Effects on GAIA: A Controlled Comparisoncited by 5
- 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Qualitycited by 5
- 2026Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluationscited by 4
- 2026Same Model, Different Harness: Different Coding-Agent Resultscited by 4
- 2026DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffoldscited by 3
- 2026The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Taskcited by 2
- 2026A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformationscited by 2
- 2026When Does Restricting a Coding Agent to executecode Help? A Regime × Agent-Design Ablationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026What makes a harness a harness: necessary and sufficient conditions for an agent harness
- 2026Finding the Right Fit: Model-Harness Interactions across Agent Tasks
- 2026Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer
- 2026Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules
- 2026Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise
- 2026How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?
- 2026What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
- 2026Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems