# Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents

arXiv: https://arxiv.org/abs/2604.11088 · HTML: https://mlsanity.com/paper/2604.11088/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 44 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 13 of 13)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle](https://mlsanity.com/paper/2510.16079/) (2025)
- [Composer 2 Technical Report](https://mlsanity.com/paper/2603.24477/) (2026)
- [A Subgoal-driven Framework for Improving Long-Horizon LLM Agents](https://mlsanity.com/paper/2603.19685/) (2026)
- [Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimization](https://mlsanity.com/paper/2512.07478/) (2025)
- [Understanding the Challenges in Iterative Generative Optimization with LLMs](https://mlsanity.com/paper/2603.23994/) (2026)

## Built on by (most cited first, 3 of 3)

- [Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills](https://mlsanity.com/paper/2609.12012/) (2026)
- [Curating Always-Loaded Context for LLM Agents: A Capacitated Assortment Model with Censored Feedback](https://mlsanity.com/paper/2610.11007/) (2026)
- [Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules](https://mlsanity.com/paper/2610.14216/) (2026)

## Abstract (arXiv)

Random rules improve a coding agent's task performance as much as expert-curated ones (both +13.8pp on a discriminative subset of SWE-bench Verified), and in our data every individually beneficial rule is a negative constraint ("do not refactor unrelated code"), while every individually harmful one is a positive directive ("follow code style"). We arrive at these findings through the first large-scale controlled study of agent rule files (textttCLAUDE.md, texttt.cursorrules, and the broader family of agent skills, plugin manifests, and persona definitions): we scrape 679 rule files (25,532 rules) from GitHub and conduct over 5,000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified. Three patterns emerge. (i) Rule polarity cleanly separates beneficial from harmful rules; we read this through the lens of potential-based reward shaping (PBRS). (ii) Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism. (iii) Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50. These findings expose a hidden reliability risk in the rapidly growing ecosystem of community-authored rules and skills, and they yield a clear principle for safer agent configuration: constrain what agents must not do, rather than prescribing what they should.