Scanning the Harness: A Repository Study of Configuration Exposures in AI Coding Agents
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
AI coding agents rely on repository instructions, skills, hooks, tool-server declarations, and subagent definitions. These artifacts distribute both behavior and access to executable dependencies, making configuration review part of the agent software supply chain. We study 3,171 public GitHub repositories: 2,660 assembled setups and 511 skill collections. Deterministic analysis, mechanical re-derivation, model-assisted review, and platform documentation checks identify six categories of configuration exposure and conformance issues. Unpinned MCP package declarations occur in 9.8% of setups, broad execution grants in 2.5%, and broad skill tool preapproval in 3.8%. Their union covers 409 setups (15.4%); among setups with MCP configuration, 24.5% contain an unpinned declaration. Including required-field and skill-format issues brings the setup rate to 17.9% and the collection rate to 6.8%. Holding the six categories fixed, contextual review changes the setup rate from 18.3% to 17.9%; rule selection explains most of the reduction from the broader candidate set. The findings identify concrete opportunities to pin dependencies, review execution preapproval, and check component conformance before distribution or use. A documented permission exception also exposes a shared error in the scanner and its mechanical audit, motivating version-specific semantic checks. The study provides reproducible evidence about repository declarations; runtime harm and detector accuracy against human ground truth remain open questions.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 6 of its 30 references that are in the map. First posted on arXiv 7 Sep 2026.
Builds on
6 of 30 references.
Closest ancestors · 2
Same direction, newest first
- 2026AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces24 Aug
- 2025Agent READMEs: An Empirical Study of Context Files for Agentic Coding17 Nov
Other references · 4
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
- 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 17
- 2026Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skillscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projects
- 2026Harness Engineering for Agentic AI Coding Tools: An Exploratory Study
- 2026A Dataset of Agentic AI Coding Tool Configurations
- 2026From Registry to Repository: How AI Agent Skills Are Written, Adapted, and Maintained
- 2026A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption
- 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software
- 2026Govern the Repository, Not the Agent: Measuring Ecosystem-Level Risk in AI-Native Software
- 2026The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol