mlsanityData to 14 Oct 2026

Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite

arXiv 2609.11987 ↗First posted 8 Sep 2026cs.AI cs.CL cs.SEcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners assume the vendor-native pairing solves more tasks. We measure that assumption with paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks. The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells. 792 of 800 planned runs were graded by an isolated oracle. Neither contrast resolves an average advantage for either harness: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, task-bootstrap 95% CI [-10.0, +7.5]) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%, CI [-4.4, +6.9]). The Opus average combines opposite strata: the native harness trails by 9.0 pp on the 61 repository tasks and leads by 23.7 pp on the 19 contest tasks (label-permutation p = 0.003). The partition was chosen after seeing the data and needs a designed replication. Correctness and completion also separate: 22 of 81 runs cancelled at the wall-clock ceiling had produced a passing patch. Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5. These are observed-usage estimates. On the Anthropic account 58 runs left no usage record, and allocating that spend to either cell would move the Opus ratio between 0.7 and 2.3, so the billed ordering is unresolved. This revision corrects an August 2026 manuscript whose cost figures rested on a usage-semantics defect in our own telemetry (Section 5.1). We release the orchestrator, grading oracle, reanalysis code and derived aggregates. The tasks stay private.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 7 of its 15 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

7 of 15 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Same Model, Different Harness: Different Coding-Agent Results26 Aug
  2. 2026The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation8 Jun
  3. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows27 May
  4. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul

Other references · 3

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Empire, Long Divided, Must Unite: Architectural Convergence in Three LLM Agent Harnesses
  2. 2026Code as Agent Harness
  3. 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality
  4. 2026An Empirical Study of Harness Design for Coding Agents
  5. 2026Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems
  6. 2026Finding the Right Fit: Model-Harness Interactions across Agent Tasks
  7. 2026Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
  8. 2026Composing Task-specific Agent Harnesses at Test Time with Reusable Primitives