How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
AI coding agents are increasingly acting as autonomous contributors by generating and submitting pull requests (PRs). However, we lack empirical evidence on how these agent-generated PRs differ from human contributions, particularly in how they modify code and describe their changes. Understanding these differences is essential for assessing their reliability and impact on development workflows. Using the MSR 2026 Mining Challenge version of the AIDev dataset, we analyze 24,014 merged Agentic PRs (440,295 commits) and 5,081 merged Human PRs (23,242 commits). We examine additions, deletions, commits, and files touched, and evaluate the consistency between PR descriptions and their diffs using lexical and semantic similarity. Agentic PRs differ substantially from Human PRs in commit count (Cliff's δ = 0.5429) and show moderate differences in files touched and deleted lines. They also exhibit slightly higher description-to-diff similarity across all measures. These findings provide a large-scale empirical characterization of how AI coding agents contribute to open source development.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 44 references that are in the map. First posted on arXiv 24 Jan 2026.
Builds on
13 of 44 references.
Closest ancestors · 4
Same direction, newest first
- 2025An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications23 Sep
- 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub18 Sep
- 2025Agentic Software Engineering: Foundational Pillars and a Research Roadmap7 Sep
- 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios16 Oct
Other references · 9
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
- 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
- 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
- 2023Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Educationcited by 38
- 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
- 2025Improved IR-based Bug Localization with Intelligent Relevance Feedbackcited by 5
- 2025An Empirical Study on Commit Message Generation using LLMs via In-Context Learningcited by 5
- 2025Refactoring-Aware Patch Integration Across Structurally Divergent Java Forkscited by 2
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHubcited by 4
- 2026AIRA: AI-Induced Risk Audit: A Structured Inspection Framework for AI-Generated Codecited by 0
- 2026Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoptioncited by 0
- 2026Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agentscited by 0
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requestscited by 0
- 2026AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprintscited by 0
- 2026Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requestscited by 0
- 2026Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests22 Sep 2026
- 2026AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprints2 Aug 2026
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests23 Jul 2026
- 2026Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents14 Jul 2026
- 2026Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoption24 Jun 2026
- 2026AIRA: AI-Induced Risk Audit: A Structured Inspection Framework for AI-Generated Code19 Apr 2026
- 2026AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub4 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
- 2026Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
- 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
- 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests
- 2026Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild
- 2026Early Adoption of Agentic Coding Tools by GitHub Projects
- 2026Let's Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests
- 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study