Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 27 references that are in the map. First posted on arXiv 28 Jul 2026.
Builds on
12 of 27 references.
Closest ancestors · 3
Same direction, newest first
- 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?12 Feb
- 2026On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents28 Jan
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
Other references · 9
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
- 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
- 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
- 2026Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprisecited by 1
- 2026From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentationcited by 0
- 2026Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Taskscited by 0
- 2026Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise24 Sep 2026
- 2026Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks7 Sep 2026
- 2026From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation20 Aug 2026
- 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model14 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2025Agent READMEs: An Empirical Study of Context Files for Agentic Coding
- 2026Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
- 2026AI Agents Do Not Fail Alone:The Context Fails First
- 2026Codified Context: Infrastructure for AI Agents in a Complex Codebase
- 2025Context Engineering for AI Agents in Open-Source Software
- 2026Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents
- 2026Mise en Place for Agentic Coding: Deliberate Preparation as Context Engineering Methodology
- 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software