# AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents arXiv: https://arxiv.org/abs/2605.13357 ยท HTML: https://mlsanity.com/paper/2605.13357/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 11 of its 17 references that are in the map. - First posted on arXiv 13 May 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 11 of 11) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023) - [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024) - [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023) - [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024) - [AIOS: LLM Agent Operating System](https://mlsanity.com/paper/2403.16971/) (2024) ## Built on by (most cited first, 20 of 20) - [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026) - [Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable](https://mlsanity.com/paper/2607.13285/) (2026) - [Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation](https://mlsanity.com/paper/2606.23743/) (2026) - [OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents](https://mlsanity.com/paper/2608.05013/) (2026) - [From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents](https://mlsanity.com/paper/2607.08028/) (2026) - [Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence](https://mlsanity.com/paper/2608.21156/) (2026) - [How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents](https://mlsanity.com/paper/2609.20474/) (2026) - [Towards an Agent Operating System - Lessons from Classical and Cloud OS](https://mlsanity.com/paper/2607.25076/) (2026) - [AI-driven Software Development: A Pragmatic Path to Agentic Development Processes](https://mlsanity.com/paper/2606.15283/) (2026) - [WARA: Toward Automated Wireless Optimization Research with Closed-Loop LLM Agents](https://mlsanity.com/paper/2608.14573/) (2026) - [Forward-Deployed Full-Stack Engineering for Autonomous Cloud MLOps](https://mlsanity.com/paper/2608.29615/) (2026) - [Task-Specified Active Metrological Inspection with Measurement-Steered VLA Manipulation and Deterministic Evidence Gating](https://mlsanity.com/paper/2609.14219/) (2026) - [CP-Agent: A Harness-Engineered Agent for Crystal Plasticity Simulation Workflows](https://mlsanity.com/paper/2609.31790/) (2026) - [AI Harness: Certification under Proposal-Conditioned Information for Foundation-Model Agents](https://mlsanity.com/paper/2609.32184/) (2026) - [Sol-H3: Recursive Self-Improvement for MiniMax-H3 Inference Acceleration on Sol-Engine across Cloud and Edge](https://mlsanity.com/paper/2609.35110/) (2026) - [Turbo Harness: Instance-Adaptive Harness Optimization](https://mlsanity.com/paper/2609.40330/) (2026) - [WAMJET: A Harness for World Action Model Acceleration](https://mlsanity.com/paper/2610.03797/) (2026) - [AECP: Artifact-Exclusive Communication Protocol for Multi-Agent Code Generation](https://mlsanity.com/paper/2610.06481/) (2026) - [NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime](https://mlsanity.com/paper/2610.10787/) (2026) - [Cross-Provider Review as a Runtime Contract for Coding Agents: A Controlled Pilot and Fault-Injection Study](https://mlsanity.com/paper/2610.10961/) (2026) ## Abstract (arXiv) Foundation models have transformed automated code generation, yet autonomous software-engineering agents remain unreliable in realistic development settings. The dominant explanation locates this gap in model capability. We propose a different locus: software-engineering capability emerges from a model-harness-environment system, in which a runtime substrate -- the harness -- mediates how a foundation-model agent observes a project, acts on it, receives feedback, and establishes that a change is complete. We formalize this substrate as an AI Harness Engineering and identify eleven component responsibilities: task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording. We operationalize the harness through a four-level ladder (H0-H3) that progressively exposes runtime support to the agent, and we propose a trace-based evaluation protocol that converts each agent run into an auditable episode package. Applied to a controlled validation task, the framework yields episode packages whose evidence structure varies systematically with harness level: lower levels produce only a final patch, higher levels produce reproduction logs, failure attributions, deterministic requirement checks, and structured verification reports. The framework reframes the central question of autonomous software engineering from whether a foundation model can produce a patch to whether the model-harness-environment system can produce a verifiably correct, attributed, and maintainable change. We outline a research program for the runtime systems that foundation-model software agents will require.