Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
Not on the map. Why?
Abstract (arXiv)
We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
20 of 33 references.
References · 20
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2024Long-form factuality in large language modelscited by 155
- 2018Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detectioncited by 55
- 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 43
- 2026AIDev: Studying AI Coding Agents on GitHubcited by 17
- 2025Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexitycited by 16
- 2025How Developers Interact with AI: A Taxonomy of Human-AI Collaboration in Software Engineeringcited by 15
- 2026Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessionscited by 14
- 2026Failure as a Process: An Anatomy of CLI Coding Agent Trajectoriescited by 13
- 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
8 more
- 2026Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wildcited by 10
- 2026What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agentscited by 5
- 2025LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Modelscited by 3
- 2026Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agentscited by 3
- 2026A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositoriescited by 2
- 2026To What Extent Does Agent-generated Code Require Maintenance? An Empirical Studycited by 2
- 2026AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?cited by 2
- 2025CoMRAT: Commit Message Rationale Analysis Toolcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
- 2026Quality and Security Signals in AI-Generated Python Refactoring Pull Requests
- 2026Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
- 2026Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests
- 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
- 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
- 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
- 2025Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios