Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five coding agents across GitHub. (RQ1) We first quantitatively characterize merged and not-merged PRs along four broad dimensions: 1) merge outcomes across task types, 2) code changes, 3) CI build results, and 4) review dynamics. We observe that tasks related to documentation, CI, and build update achieve the highest merge success, whereas performance and bug-fix tasks perform the worst. Not-merged PRs tend to involve larger code changes, touch more files, and often do not pass the project's CI/CD pipeline validation. (RQ2) To further investigate why some agentic PRs are not merged, we qualitatively analyze 600 PRs to derive a hierarchical taxonomy of rejection patterns. This analysis complements the quantitative findings in RQ1 by uncovering rejection reasons not captured by quantitative metrics, including lack of meaningful reviewer engagement, duplicate PRs, unwanted feature implementations, and agent misalignment. Together, our findings highlight key socio-technical and human-AI collaboration factors that are critical to improving the success of future agentic workflows.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 31 references that are in the map. First posted on arXiv 21 Jan 2026.
Builds on
13 of 31 references.
Closest ancestors · 1
Same direction, newest first
- 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories31 Oct
Other references · 12
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
- 2023Large Language Models for Test-Free Fault Localizationcited by 67
- 2023A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localizationcited by 50
- 2019Does Code Quality Affect Pull Request Acceptance? An empirical studycited by 9
- 2025Do LLMs Consider Security? An Empirical Study on Responses to Programming Questionscited by 8
- 2025Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challengescited by 6
- 2025Hierarchical Knowledge Injection for Improving LLM-based Program Repaircited by 6
- 2025What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issuescited by 5
- 2019Analyzing the Fine Structure of Distributionscited by 4
- 2023Repeated Builds During Code Review: An Empirical Study of the OpenStack Communitycited by 2
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026Wink: Recovering from Misbehaviors in Coding Agentscited by 11
- 2026How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessionscited by 11
- 2026Will It Survive? Deciphering the Fate of AI-Generated Code in Open Sourcecited by 3
- 2026PROJECTMEM: A Local-First, Event-Sourced Memory and Judgment Layer for AI Coding Agentscited by 3
- 2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Reviewcited by 1
- 2026Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoptioncited by 0
- 2026Early Adoption of Agentic Coding Tools by GitHub Projectscited by 0
- 2026A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communitiescited by 0
- 2026AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprintscited by 0
- 2026From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentationcited by 0
- 2026From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation20 Aug 2026
- 2026AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprints2 Aug 2026
- 2026A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities29 Jul 2026
- 2026Early Adoption of Agentic Coding Tools by GitHub Projects15 Jul 2026
- 2026Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoption24 Jun 2026
- 2026PROJECTMEM: A Local-First, Event-Sourced Memory and Judgment Layer for AI Coding Agents10 Jun 2026
- 2026How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions28 May 2026
- 2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review17 May 2026
- 2026Wink: Recovering from Misbehaviors in Coding Agents19 Feb 2026
- 2026Will It Survive? Deciphering the Fate of AI-Generated Code in Open Source23 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
- 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
- 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
- 2026Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests
- 2026Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study
- 2026AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub
- 2026Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
- 2026Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild