Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
AI coding agents increasingly submit pull requests (Agentic-PRs) to open-source repositories, yet their performance is commonly assessed using merge and rejection outcomes alone. We hypothesized that these outcome labels do not reliably reflect agent capability without considering review interactions. To test this, we conducted a decision-oriented analysis of 11,048 closed Agentic Pull Requests, refined to 9,799 human-reviewed PRs, and manually inspected 717 representative cases to recover decision rationale from interaction artifacts. We found that rejection outcomes substantially overstate agent error: only 35.7% of rejected PRs reflected clear agentic failures, while 31.2% were driven by workflow constraints and 33.1% lacked observable decision rationale. Among merged PRs, 15.4% required explicit reviewer involvement through feedback or direct commits, and 5.5% showed no visible interaction trace. We further observed systematic differences across agents, with Copilot and Devin more often embedded in reviewer-mediated workflows, while Codex and Cursor PRs were typically merged with minimal interaction. These results reject the assumption that PR outcomes alone capture agent performance and demonstrate the need for interaction-aware evaluation grounded in review behavior.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 7 of its 22 references that are in the map. First posted on arXiv 21 May 2026.
Builds on
7 of 22 references.
Closest ancestors · 1
Same direction, newest first
Other references · 6
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
- 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 264
- 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
- 2025Human and Machine: How Software Engineers Perceive and Engage with AI-Assisted Code Reviews Compared to Their Peerscited by 8
- 2025Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflowscited by 6
- 2021Applying Inter-rater Reliability and Agreement in Grounded Theory Studies in Software Engineeringcited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Codecited by 0
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requestscited by 0
- 2026Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requestscited by 0
- 2026Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests22 Sep 2026
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests23 Jul 2026
- 2026Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code10 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
- 2026Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
- 2026Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents
- 2026Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests
- 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
- 2026Humans Integrate, Agents Fix: How Agent-Authored Pull Requests Are Referenced in Practice
- 2026When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests
- 2026AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub