# Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase

arXiv: https://arxiv.org/abs/2609.29744 · HTML: https://mlsanity.com/paper/2609.29744/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection](https://mlsanity.com/paper/1809.08193/) (2018)
- [SWE-chat: Coding Agent Interactions From Real Users in the Wild](https://mlsanity.com/paper/2604.20779/) (2026)
- [AIDev: Studying AI Coding Agents on GitHub](https://mlsanity.com/paper/2602.09185/) (2026)
- [Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity](https://mlsanity.com/paper/2508.21634/) (2025)
- [How Developers Interact with AI: A Taxonomy of Human-AI Collaboration in Software Engineering](https://mlsanity.com/paper/2501.08774/) (2025)
- [Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions](https://mlsanity.com/paper/2604.00436/) (2026)
- [Failure as a Process: An Anatomy of CLI Coding Agent Trajectories](https://mlsanity.com/paper/2607.09510/) (2026)
- [TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems](https://mlsanity.com/paper/2605.26563/) (2026)
- [Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild](https://mlsanity.com/paper/2603.28592/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models](https://mlsanity.com/paper/2504.01404/) (2025)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositories](https://mlsanity.com/paper/2603.27130/) (2026)
- [To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study](https://mlsanity.com/paper/2605.06464/) (2026)
- [AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?](https://mlsanity.com/paper/2607.03332/) (2026)
- [CoMRAT: Commit Message Rationale Analysis Tool](https://mlsanity.com/paper/2506.10986/) (2025)

## Abstract (arXiv)

We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.