How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents
New in LLM agents and tool use; no direction yet. LLM agents and tool use
Abstract (arXiv)
Agent harnesses supply planning guidance, organize execution, and check completion. We study how these components affect success, erroneous acceptance, and cost in two Retail experiments and an Airline pilot in τ2-bench. The primary comparison pairs prewritten task-specific plans (Fixed) with shuffled policy text matched in word count (Sham), isolating the contribution of guidance content. Across 265 matched cells, Fixed improves oracle-verified success by 7.17 percentage points (90% task-clustered bootstrap interval, 1.15--13.36 points), with gains concentrated in higher-complexity tasks. A read-only terminal verifier rejects 61% of Retail oracle-invalid episodes while withholding 17% of correct ones, at less than one cent of additional cost per episode. Which component matters more depends on the loss assigned to erroneous acceptance: at low liability the planning gain dominates; at high liability the verifier's avoided false passes dominate---and a standalone verifier captures nearly all the false-pass benefit of the full planning-plus-verification stack at a fraction of its cost.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 11 of its 57 references that are in the map. First posted on arXiv 17 Sep 2026.
Builds on
11 of 57 references.
References · 11
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
- 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
- 2022Solving math word problems with process- and outcome-based feedbackcited by 901
- 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 20
- 2026AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agentscited by 20
- 2024On the Brittle Foundations of ReAct Prompting for Agentic Large Language Modelscited by 17
- 2026From Confident Closing to Silent Failure: Characterizing False Success in LLM Agentscited by 15
- 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
- 2026Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agentscited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026How Much Heavy Lifting Can an Agent Harness Do?: Measuring the LLM's Residual Role in a Planning Agent
- 2026Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails
- 2026Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints
- 2026Set-shifting Behavioral Test for Harnessed Agents
- 2026Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- 2026Dynamic Harness Search: Building Multi-Agent Systems Per-Query via Prediction
- 2026The Interplay of Harness Design and Post-Training in LLM Agents
- 2026Stop Comparing LLM Agents Without Disclosing the Harness