mlsanityData to 7 Oct 2026

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

arXiv 2607.27250 ↗First posted 28 Jul 2026cs.SE cs.AIcited by 4

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 27 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

12 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?12 Feb
  2. 2026On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents28 Jan
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 9

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  9. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent READMEs: An Empirical Study of Context Files for Agentic Coding
  2. 2026Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
  3. 2026AI Agents Do Not Fail Alone:The Context Fails First
  4. 2026Codified Context: Infrastructure for AI Agents in a Complex Codebase
  5. 2025Context Engineering for AI Agents in Open-Source Software
  6. 2026Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents
  7. 2026Mise en Place for Agentic Coding: Deliberate Preparation as Context Engineering Methodology
  8. 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software