# Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub

arXiv: https://arxiv.org/abs/2601.15195 · HTML: https://mlsanity.com/paper/2601.15195/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 13 of 13)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Large Language Models for Test-Free Fault Localization](https://mlsanity.com/paper/2310.01726/) (2023)
- [A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization](https://mlsanity.com/paper/2308.05487/) (2023)
- [Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories](https://mlsanity.com/paper/2511.00197/) (2025)
- [Does Code Quality Affect Pull Request Acceptance? An empirical study](https://mlsanity.com/paper/1908.09321/) (2019)
- [Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions](https://mlsanity.com/paper/2502.14202/) (2025)
- [Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges](https://mlsanity.com/paper/2506.04418/) (2025)
- [What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issues](https://mlsanity.com/paper/2506.22390/) (2025)
- [Hierarchical Knowledge Injection for Improving LLM-based Program Repair](https://mlsanity.com/paper/2506.24015/) (2025)
- [Analyzing the Fine Structure of Distributions](https://mlsanity.com/paper/1908.06081/) (2019)
- [Repeated Builds During Code Review: An Empirical Study of the OpenStack Community](https://mlsanity.com/paper/2308.10078/) (2023)

## Built on by (most cited first, 13 of 13)

- [Wink: Recovering from Misbehaviors in Coding Agents](https://mlsanity.com/paper/2602.17037/) (2026)
- [How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions](https://mlsanity.com/paper/2605.29442/) (2026)
- [Will It Survive? Deciphering the Fate of AI-Generated Code in Open Source](https://mlsanity.com/paper/2601.16809/) (2026)
- [PROJECTMEM: A Local-First, Event-Sourced Memory and Judgment Layer for AI Coding Agents](https://mlsanity.com/paper/2606.12329/) (2026)
- [Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review](https://mlsanity.com/paper/2605.17548/) (2026)
- [A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities](https://mlsanity.com/paper/2607.26819/) (2026)
- [Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoption](https://mlsanity.com/paper/2606.26289/) (2026)
- [3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse](https://mlsanity.com/paper/2607.07980/) (2026)
- [Early Adoption of Agentic Coding Tools by GitHub Projects](https://mlsanity.com/paper/2607.14037/) (2026)
- [AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprints](https://mlsanity.com/paper/2608.00966/) (2026)
- [From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation](https://mlsanity.com/paper/2608.20195/) (2026)
- [Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests](https://mlsanity.com/paper/2610.12849/) (2026)
- [SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents](https://mlsanity.com/paper/2610.18206/) (2026)

## Abstract (arXiv)

AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five coding agents across GitHub. (RQ1) We first quantitatively characterize merged and not-merged PRs along four broad dimensions: 1) merge outcomes across task types, 2) code changes, 3) CI build results, and 4) review dynamics. We observe that tasks related to documentation, CI, and build update achieve the highest merge success, whereas performance and bug-fix tasks perform the worst. Not-merged PRs tend to involve larger code changes, touch more files, and often do not pass the project's CI/CD pipeline validation. (RQ2) To further investigate why some agentic PRs are not merged, we qualitatively analyze 600 PRs to derive a hierarchical taxonomy of rejection patterns. This analysis complements the quantitative findings in RQ1 by uncovering rejection reasons not captured by quantitative metrics, including lack of meaningful reviewer engagement, duplicate PRs, unwanted feature implementations, and agent misalignment. Together, our findings highlight key socio-technical and human-AI collaboration factors that are critical to improving the success of future agentic workflows.