mlsanityData to 7 Oct 2026

Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub

arXiv 2601.15195 ↗First posted 21 Jan 2026cs.SE cs.AIcited by 10

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five coding agents across GitHub. (RQ1) We first quantitatively characterize merged and not-merged PRs along four broad dimensions: 1) merge outcomes across task types, 2) code changes, 3) CI build results, and 4) review dynamics. We observe that tasks related to documentation, CI, and build update achieve the highest merge success, whereas performance and bug-fix tasks perform the worst. Not-merged PRs tend to involve larger code changes, touch more files, and often do not pass the project's CI/CD pipeline validation. (RQ2) To further investigate why some agentic PRs are not merged, we qualitatively analyze 600 PRs to derive a hierarchical taxonomy of rejection patterns. This analysis complements the quantitative findings in RQ1 by uncovering rejection reasons not captured by quantitative metrics, including lack of meaningful reviewer engagement, duplicate PRs, unwanted feature implementations, and agent misalignment. Together, our findings highlight key socio-technical and human-AI collaboration factors that are critical to improving the success of future agentic workflows.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 31 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

13 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories31 Oct

Other references · 12

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  4. 2023Large Language Models for Test-Free Fault Localizationcited by 67
  5. 2023A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localizationcited by 50
  6. 2019Does Code Quality Affect Pull Request Acceptance? An empirical studycited by 9
  7. 2025Do LLMs Consider Security? An Empirical Study on Responses to Programming Questionscited by 8
  8. 2025Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challengescited by 6
  9. 2025Hierarchical Knowledge Injection for Improving LLM-based Program Repaircited by 6
  10. 2025What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issuescited by 5
  11. 2019Analyzing the Fine Structure of Distributionscited by 4
  12. 2023Repeated Builds During Code Review: An Empirical Study of the OpenStack Communitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
  2. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
  3. 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
  4. 2026Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests
  5. 2026Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study
  6. 2026AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub
  7. 2026Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
  8. 2026Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild