mlsanityData to 14 Oct 2026

What makes a harness a harness: necessary and sufficient conditions for an agent harness

arXiv 2606.10106 ↗First posted 8 Jun 2026cs.SE cs.AIcited by 6

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

The term agent harness now circulates widely in software engineering with generative artificial intelligence. It names the layer that wraps a language model and turns it into a coding agent able to act on a repository. The usage is loose and polysemous. Sometimes the term denotes the whole product (Claude Code, Codex CLI); sometimes it denotes the evaluation scaffold that runs an agent against tasks (the SWE-bench harness); sometimes it gets conflated with an agent framework, an SDK, an IDE plugin, or an orchestrator. What is missing is a reference definition that works as an instrument, one that includes and excludes cases consistently. We build that definition through a conceptual analysis that combines works with persistent identifiers and primary grey-literature sources, such as official documentation, glossaries, and engineering reports. We reconstruct the genealogy of the term, from the horse's tack to the classic test harness, to the machine-learning evaluation harness, and finally to the agent harness. We then propose a constitutive definition that states the necessary and sufficient conditions for a system to be an agent harness, we operationalize it as an inclusion and exclusion test, and we draw the boundary of the concept against an agent framework, an agent SDK, an IDE plugin, an eval harness, and an orchestrator. We apply the definition to six real harnesses (Claude Code, Codex CLI, Aider, Cline, OpenHands, and SWE-agent) and to deliberate edge cases; the test includes and excludes consistently. We close with a research agenda organized by design tension axes. The contribution is an operational definition of agent harness, with a shared vocabulary, able to guide engineering practice and the scientific comparison of agentic systems.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 63 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

54 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  2. 2024MASAI: Modular Architecture for Software-engineering AI Agents17 Jun
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
  4. 2024Executable Code Actions Elicit Better LLM Agents1 Feb
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug
  6. 2023Self-collaboration Code Generation via ChatGPT15 Apr

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  12. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
36 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  12. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  13. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  14. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  15. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  16. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  17. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  18. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  19. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  20. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  21. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  22. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  23. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  24. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  25. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  26. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  27. 2025Survey on Evaluation of LLM-based Agentscited by 204
  28. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  29. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
  30. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  31. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  32. 2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 121
  33. 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
  34. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
  35. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
  36. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
  2. 2026Harness Engineering as Categorical Architecture
  3. 2026Code as Agent Harness
  4. 2026An Empirical Study of Harness Design for Coding Agents
  5. 2026AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents
  6. 2026HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
  7. 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality
  8. 2026Finding the Right Fit: Model-Harness Interactions across Agent Tasks