# Builder, Defender, Breaker: Measurable Independence and Bounded Autonomy When Generative Models Build, Defend and Test Software

arXiv: https://arxiv.org/abs/2607.03215 · HTML: https://mlsanity.com/paper/2607.03215/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 19 of 19)

- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://mlsanity.com/paper/2506.06941/) (2025)
- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Adversarial Examples that Fool both Computer Vision and Time-Limited Humans](https://mlsanity.com/paper/1802.08195/) (2018)
- [TrojanPuzzle: Covertly Poisoning Code-Suggestion Models](https://mlsanity.com/paper/2301.02344/) (2023)
- [Great Models Think Alike and this Undermines AI Oversight](https://mlsanity.com/paper/2502.04313/) (2025)
- [How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models](https://mlsanity.com/paper/2404.18353/) (2024)
- [N-Version Programming with Coding Agents](https://mlsanity.com/paper/2606.20158/) (2026)
- [A Systematic Methodology for Evaluating Failure Independence in LLM-Generated Code](https://mlsanity.com/paper/2607.02808/) (2026)

## Abstract (arXiv)

Generative models now write application code, harden and monitor it, and probe it for exploitable flaws, so that one family of models increasingly plays builder, defender and breaker at once. The prevailing view treats full autonomy as the natural end point of assistance. This article argues for a narrower and more defensible position than a blanket requirement for human oversight. We define the shared generative substrate as the set of upstream dependencies (training corpus, model family, alignment procedure, vendor, toolchain) that induce correlated errors, and we define independence as a measurable property of pairs of lifecycle roles rather than an attribute of any participant. A coincident-failure model in the tradition of Eckhardt and Lee shows why organisational independence, the proxy on which verification standards have relied, no longer implies statistical independence once roles share a substrate; it also shows that heterogeneous models, deterministic analysers and formal verification can restore independence for the fault classes they cover. What remains for humans is specific: authority over the specification against which every machine oracle is judged, accountability that current governance instruments do not permit to be delegated, and last-resort authority to halt. We translate this into an operational framework with five autonomy levels, three distinct human roles and five decision criteria (consequence, reversibility, time-criticality, verifiability and adversarial exposure), apply it to build, defend and test actions, and specify how independence and oversight effectiveness can be measured. The central claims are stated as testable hypotheses with a protocol for refuting them.