Harness Engineering for Agentic AI Coding Tools: An Exploratory Study
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Agentic AI coding tools increasingly automate software development tasks. Developers can configure these tools through versioned repository-level artifacts such as Markdown and JSON files. We present a systematic analysis of configuration mechanisms for agentic AI coding tools, covering Claude Code, GitHub Copilot, Cursor, Gemini, and Codex. We identify eight configuration mechanisms spanning from static context to executable and external integrations and, in an empirical study of 2,853 GitHub repositories, examine whether and how they are adopted, with a detailed analysis of Context Files, Skills, and Subagents. First, Context Files dominate the configuration landscape and are often the sole mechanism in a repository, with AGENTS.md emerging as an interoperable standard across tools. Second, few repositories adopt advanced mechanisms such as Skills and Subagents. Skills predominantly rely on static instructions rather than executable scripts. Third, distinct configuration practices are forming around different tools, with Claude Code users employing the broadest range of mechanisms. These findings establish an empirical baseline for understanding how developers configure agentic tools, suggest that AGENTS.md serves as a natural starting point, and motivate longitudinal and experimental research on how configuration strategies evolve and affect agent performance.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 32 references that are in the map. First posted on arXiv 16 Feb 2026.
Builds on
13 of 32 references.
Closest ancestors · 5
Same direction, newest first
- 2026On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents28 Jan
- 2026Agentic Much? Adoption of Coding Agents on GitHub26 Jan
- 2025Agent READMEs: An Empirical Study of Context Files for Agentic Coding17 Nov
- 2025Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projects12 Nov
- 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects6 Nov
Other references · 8
- 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
- 2024Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forwardcited by 44
- 2026Meta Context Engineering via Agentic Skill Evolutioncited by 42
- 2021Sampling Projects in GitHub for MSR Studiescited by 20
- 2025Context Engineering for AI Agents in Open-Source Softwarecited by 14
- 2025Beyond the Prompt: An Empirical Study of Cursor Rulescited by 7
- 2026Structured Context Engineering for File-Native Agentic Systems: Evaluating Schema Accuracy, Format Effectiveness, and Multi-File Navigation at Scalecited by 4
- 2025Prompts as Software Engineering Artifacts: A Research Agenda and Preliminary Findingscited by 2
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architecturescited by 15
- 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
- 2026Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Developmentcited by 2
- 2026Context Rot in AI-Assisted Software Development: Repurposing Documentation Consistency for AI Configuration Artifactscited by 2
- 2026Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational studycited by 2
- 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligencecited by 1
- 2026EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?cited by 1
- 2026Beyond Human-Readable: Rethinking Software Engineering Conventions for the Agentic Development Eracited by 0
- 2026The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocolcited by 0
- 2026OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systemscited by 0
- 2026EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?3 Sep 2026
- 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence21 Aug 2026
- 2026Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study2 Jul 2026
- 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Design14 Jun 2026
- 2026Context Rot in AI-Assisted Software Development: Repurposing Documentation Consistency for AI Configuration Artifacts8 Jun 2026
- 2026The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol2 Jun 2026
- 2026Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development19 May 2026
- 2026OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systems16 Apr 2026
- 2026Beyond Human-Readable: Rethinking Software Engineering Conventions for the Agentic Development Era8 Apr 2026
- 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures3 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2026A Dataset of Agentic AI Coding Tool Configurations
- 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software
- 2026Scanning the Harness: A Repository Study of Configuration Exposures in AI Coding Agents
- 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
- 2025On the Use of Agentic Coding Manifests: An Empirical Study of Claude Code
- 2026Code as Agent Harness
- 2025Agentic Refactoring: An Empirical Study of AI Coding Agents
- 2026Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows