# Stop Comparing LLM Agents Without Disclosing the Harness

arXiv: https://arxiv.org/abs/2605.23950 · HTML: https://mlsanity.com/paper/2605.23950/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 25 of 25)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses](https://mlsanity.com/paper/2604.25850/) (2026)
- [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025)
- [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://mlsanity.com/paper/2510.11977/) (2025)
- [AutoHarness: improving LLM agents by automatically synthesizing a code harness](https://mlsanity.com/paper/2603.03329/) (2026)
- [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025)
- [APEX-Agents](https://mlsanity.com/paper/2601.14242/) (2026)
- [Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction](https://mlsanity.com/paper/2512.04987/) (2025)
- [Agent psychometrics: Task-level performance prediction in agentic coding benchmarks](https://mlsanity.com/paper/2604.00594/) (2026)

## Built on by (most cited first, 30 of 34)

- [RHO: Your Coding Agent is Secretly a Roboticist](https://mlsanity.com/paper/2606.16458/) (2026)
- [Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?](https://mlsanity.com/paper/2608.04828/) (2026)
- [HarnessOpt-Bench: Evaluating LLMs at Harness Optimization](https://mlsanity.com/paper/2608.06301/) (2026)
- [What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents](https://mlsanity.com/paper/2609.04518/) (2026)
- [DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum](https://mlsanity.com/paper/2609.19801/) (2026)
- [GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes](https://mlsanity.com/paper/2606.28514/) (2026)
- [RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescue](https://mlsanity.com/paper/2607.01213/) (2026)
- [A Control System, a Dataset, and a Recipe for Making Frozen LLM Agents Learn a Domain](https://mlsanity.com/paper/2607.25415/) (2026)
- [CHILL-Harness: Counterfactual Harness Learning for Efficient Reasoning in Long-Horizon Agents](https://mlsanity.com/paper/2607.25825/) (2026)
- [QuoteBench: How Matched Scores Can Hide Command-Path Failures](https://mlsanity.com/paper/2608.13547/) (2026)
- [Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?](https://mlsanity.com/paper/2609.28908/) (2026)
- [OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime](https://mlsanity.com/paper/2609.38345/) (2026)
- [Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI](https://mlsanity.com/paper/2607.02873/) (2026)
- [Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents](https://mlsanity.com/paper/2607.11433/) (2026)
- [Context Assembly as the Controlled Variable: A Control-Theoretic View of Harness Policies for Frozen LLM Agents](https://mlsanity.com/paper/2607.25408/) (2026)
- [The Empire, Long Divided, Must Unite: Architectural Convergence in Three LLM Agent Harnesses](https://mlsanity.com/paper/2608.23953/) (2026)
- [Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification](https://mlsanity.com/paper/2609.16093/) (2026)
- [ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts](https://mlsanity.com/paper/2609.18844/) (2026)
- [CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine](https://mlsanity.com/paper/2609.23142/) (2026)
- [SRHarness: A Harness for Agentic Symbolic Regression](https://mlsanity.com/paper/2609.35501/) (2026)
- [Simple Agentic Memory for Generalist Robot Policies](https://mlsanity.com/paper/2609.36595/) (2026)
- [When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoning](https://mlsanity.com/paper/2609.39142/) (2026)
- [K-Dense BYOK: An Open-Source AI Research Assistant That Runs Locally and Keeps a Hash-Chained Lab Notebook](https://mlsanity.com/paper/2610.00074/) (2026)
- [Deny Without Disabling: Authorization-Paired Evaluation and Control for Multi-Agent Systems](https://mlsanity.com/paper/2610.00371/) (2026)
- [Harness Annealing: Learning to Act with Less External Control](https://mlsanity.com/paper/2610.01235/) (2026)
- [Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks](https://mlsanity.com/paper/2610.02001/) (2026)
- [What Does a Harness Buy? Tokens, Mostly](https://mlsanity.com/paper/2610.04433/) (2026)
- [Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild](https://mlsanity.com/paper/2610.04921/) (2026)
- [MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks](https://mlsanity.com/paper/2610.05778/) (2026)
- [Attention Tax, Handoff Tax: A Stylised Model of When Multi-Agent LLM Systems Help](https://mlsanity.com/paper/2610.06069/) (2026)

## Abstract (arXiv)

This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps. We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines. First, a control-theoretic formalization treats the harness as the controller of a closed-loop dynamical system and the LLM as the stochastic policy it governs, which explains why small harness changes can produce performance shifts that exceed those obtained by substituting one model for another. Second, published benchmarks, industry deployments, and a controlled variance decomposition show that harness-induced variance can substantially exceed model-induced variance, including cases of model ranking reversal. Third, we propose a harness-aware evaluation framework with a disclosure standard and a variance decomposition protocol. Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.