mlsanityData to 13 Oct 2026

Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables

arXiv 2605.10039 ↗First posted 11 May 2026cs.SE cs.CLcited by 3

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Frontier coding agents read configuration files (CLAUDE.md, AGENTS.md, Cursor Rules) at session start and are expected to follow the conventions inside them. Practitioners assume that structural choices (file size, instruction position, file architecture, contradictions in adjacent files) measurably affect adherence. We report a systematic factorial study of these choices using four manipulated variables, measuring compliance with a trivial target annotation across 1,650 Claude Code CLI sessions (16,050 function-level observations) on two TypeScript codebases, three frontier models (primarily Sonnet 4.6, with Opus 4.6 as a CLI-matched cross-model check and Opus 4.7 reported descriptively under a CLI-version confound), and five coding tasks. We use mixed-effects models with a Bayesian companion. None of the four structural variables or three two-way interactions produces a detectable contrast after multiple-testing correction. Size and conflict nulls are supported by affirmative-null Bayes factors (BF10 between 0.05 and 0.10); position and architecture nulls are failures to reject without Bayes-factor support. The largest effect we measured is within-session: each additional function the agent generates is associated with approximately 5.6% lower odds of compliance per step (OR = 0.944) within the session-length range we tested, though the relationship is non-monotonic rather than a constant per-step effect. This reproduces on a second TypeScript codebase and on Opus 4.6 at matched configuration; it was identified during analysis rather than pre-specified. Within the conditions tested, file-structure variables did not produce detectable contrasts; compliance varies systematically between coding tasks and across each session's sequence of generated functions.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 6 of its 6 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

6 of 6 references.

References · 6

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,302
  2. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 33
  3. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 32
  4. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  5. 2026Structured Context Engineering for File-Native Agentic Systems: Evaluating Schema Accuracy, Format Effectiveness, and Multi-File Navigation at Scalecited by 5
  6. 2026Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalizationcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
  2. 2026Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
  3. 2025Agent READMEs: An Empirical Study of Context Files for Agentic Coding
  4. 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents
  5. 2026Delivery, Not Storage: Cue-Anchored Working Memory as a Harness Property for Coding Agents
  6. 2026Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents
  7. 2026Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents
  8. 2026When and How Context Rot Appears in Coding Agents: A White-Box Study of Agent Skills in Code Auditing