mlsanityData to 14 Oct 2026

It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers

arXiv 2605.26731 ↗First posted 26 May 2026cs.AI cs.CLcited by 1

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models need proportionally less structural guidance -- together implying a monotone inverse relationship between model capability tier and optimal harness complexity. We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions (light, balanced, strict) on HEAT-24, a 24-task synthetic benchmark with git-based workspace verification. Our results refute the monotone inverse relationship on two fronts. First, for the frontier chat model evaluated (Gemini 2.5 Flash), increased harness verbosity lowers VTSR by 29-38 percentage points -- a harness-complexity paradox. Second, for the frontier reasoning model evaluated (Qwen3.5-122B, extended thinking enabled), strict harness achieves the highest VTSR (91.7%) and the lowest latency, the opposite of the prediction. Within the constrained tier, a 2B model (Gemma4:e2B) matches strong-open-tier stability at 91.7% across all harnesses. Because each tier is represented by a single model in this study, these results should be interpreted as model-specific observations; harness sensitivity appears non-monotone across the models evaluated, and depends critically on model type (chat vs. reasoning). We introduce a six-label failure taxonomy showing that format_violation dominates capable-model failures while wrong_file dominates low-capability failures, and we derive practical tier-aware harness selection guidelines.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 11 of its 15 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

11 of 15 references.

References · 11

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  5. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  6. 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 121
  7. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  8. 2025Decoupling Task-Solving and Output Formatting in LLM Generationcited by 9
  9. 2025Decomposing LLM Self-Correction: The Accuracy-Correction Paradox and Error Depth Hypothesiscited by 8
  10. 2026Brevity Constraints Reverse Performance Hierarchies in Language Modelscited by 4
  11. 2025You Don't Need Prompt Engineering Anymore: The Prompting Inversioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Stop Comparing LLM Agents Without Disclosing the Harness
  2. 2026Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
  3. 2026Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation
  4. 2026Rethinking the Evaluation of Harness Evolution for Agents
  5. 2026Self-Harness: Harnesses That Improve Themselves
  6. 2026HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
  7. 2026The Interplay of Harness Design and Post-Training in LLM Agents
  8. 2026HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems