What makes a harness a harness: necessary and sufficient conditions for an agent harness
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
The term agent harness now circulates widely in software engineering with generative artificial intelligence. It names the layer that wraps a language model and turns it into a coding agent able to act on a repository. The usage is loose and polysemous. Sometimes the term denotes the whole product (Claude Code, Codex CLI); sometimes it denotes the evaluation scaffold that runs an agent against tasks (the SWE-bench harness); sometimes it gets conflated with an agent framework, an SDK, an IDE plugin, or an orchestrator. What is missing is a reference definition that works as an instrument, one that includes and excludes cases consistently. We build that definition through a conceptual analysis that combines works with persistent identifiers and primary grey-literature sources, such as official documentation, glossaries, and engineering reports. We reconstruct the genealogy of the term, from the horse's tack to the classic test harness, to the machine-learning evaluation harness, and finally to the agent harness. We then propose a constitutive definition that states the necessary and sufficient conditions for a system to be an agent harness, we operationalize it as an inclusion and exclusion test, and we draw the boundary of the concept against an agent framework, an agent SDK, an IDE plugin, an eval harness, and an orchestrator. We apply the definition to six real harnesses (Claude Code, Codex CLI, Aider, Cline, OpenHands, and SWE-agent) and to deliberate edge cases; the test includes and excludes consistently. We close with a research agenda organized by design tension axes. The contribution is an operational definition of agent harness, with a shared vocabulary, able to guide engineering practice and the scientific comparison of agentic systems.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 63 references that are in the map. First posted on arXiv 8 Jun 2026.
Builds on
54 of 63 references.
Closest ancestors · 6
Same direction, newest first
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
- 2024MASAI: Modular Architecture for Software-engineering AI Agents17 Jun
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
- 2024Executable Code Actions Elicit Better LLM Agents1 Feb
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug
- 2023Self-collaboration Code Generation via ChatGPT15 Apr
Other references · 48
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
36 more
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
- 2023A Survey of Large Language Modelscited by 2,576
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2023A Survey on Evaluation of Large Language Modelscited by 1,242
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
- 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
- 2025Survey on Evaluation of LLM-based Agentscited by 204
- 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
- 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
- 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
- 2025A Survey of Context Engineering for Large Language Modelscited by 126
- 2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 121
- 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
- 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
- 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
- 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systemscited by 6
- 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligencecited by 3
- 2026What makes prompts a graph: necessary and sufficient conditions for prompt graph engineeringcited by 1
- 2026From Prompt to Harness: Coderlet from Scratchcited by 0
- 2026Agents Are Systems, Not Models: Rethinking Agentic Evaluationcited by 0
- 2026Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentscited by 0
- 2026Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents14 Oct 2026
- 2026Agents Are Systems, Not Models: Rethinking Agentic Evaluation1 Oct 2026
- 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence21 Aug 2026
- 2026From Prompt to Harness: Coderlet from Scratch10 Aug 2026
- 2026What makes prompts a graph: necessary and sufficient conditions for prompt graph engineering30 Jul 2026
- 2026Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems15 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- 2026Harness Engineering as Categorical Architecture
- 2026Code as Agent Harness
- 2026An Empirical Study of Harness Design for Coding Agents
- 2026AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents
- 2026HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality
- 2026Finding the Right Fit: Model-Harness Interactions across Agent Tasks