On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are accepted in real-world projects. In this paper, we empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 diverse open-source projects. Our analysis reveals that developers tend to rely on agents for tasks such as refactoring, documentation, and testing. The results indicate that 83.8% of these agent-assisted PRs are eventually accepted and merged by project maintainers, with 54.9% of the merged PRs are integrated without further modification. The remaining 45.1% require additional changes benefit from human revisions, especially for bug fixes, documentation, and adherence to project-specific standards. These findings suggest that while agent-assisted PRs are largely acceptable, they still benefit from human oversight and refinement.
Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 47 references that are in the map. First posted on arXiv 18 Sep 2025.
Builds on
33 of 47 references.
Closest ancestors · 4
Same direction, newest first
- 2024Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review18 Sep
- 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead7 Apr
- 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repair25 Mar
- 2023Self-collaboration Code Generation via ChatGPT15 Apr
Other references · 29
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
- 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
- 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
- 2025Model Context Protocol (MCP) at First Glance: Studying the Security and Maintainability of MCP Serverscited by 62
- 2021Towards Automating Code Review Activitiescited by 54
- 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
- 2023No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPTcited by 45
- 2023Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Studycited by 44
- 2023On the Effectiveness of Large Language Models in Domain-Specific Code Generationcited by 44
- 2024Automatic Programming: Large Language Models and Beyondcited by 35
- 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
17 more
- 2021How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Studycited by 29
- 2023An Empirical Comparison of Pre-Trained Models of Source Codecited by 25
- 2024On Evaluating the Efficiency of Source Code Generated by LLMscited by 19
- 2024Exploring the Capabilities of LLMs for Code Change Related Taskscited by 18
- 2024Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Studycited by 16
- 2024The Role of Generative AI in Software Development Productivity: A Pilot Case Studycited by 14
- 2024Empowering Agile-Based Generative Software Development through Human-AI Teamworkcited by 12
- 2023The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Productscited by 10
- 2025Accountability in Code Review: The Role of Intrinsic Drivers and the Impact of LLMscited by 8
- 2024Fight Fire with Fire: How Much Can We Trust ChatGPT on Source Code-Related Tasks?cited by 6
- 2023The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Taskscited by 5
- 2022AutoPRTitle: A Tool for Automatic Pull Request Title Generationcited by 4
- 2023Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testingcited by 4
- 2024Generative AI for Test Driven Development: Preliminary Resultscited by 3
- 2024Are Large Language Models a Threat to Programming Platforms? An Exploratory Studycited by 3
- 2025Intention is All You Need: Refining Your Code from Your Intentioncited by 3
- 2025Enhancing Code Generation via Bidirectional Comment-Level Mutual Groundingcited by 3
Built on by
Citing papers per year: 2025: 8; 2026: 45.
53 papers.
Sort:
By area:
- 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
- 2025Agentic Refactoring: An Empirical Study of AI Coding Agentscited by 17
- 2025Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projectscited by 13
- 2026AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Developmentcited by 10
- 2025Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learningcited by 9
- 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requestscited by 7
- 2026Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHubcited by 6
- 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agentscited by 6
- 2026What to Cut? Predicting Unnecessary Methods in Agentic Code Generationcited by 5
- 2025How Do Agents Perform Code Optimization? An Empirical Studycited by 4
- 2026Understanding Dominant Themes in Reviewing Agentic AI-authored Codecited by 4
- 2026Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requestscited by 3
38 more
- 2026Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agentscited by 3
- 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Studycited by 3
- 2026Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agentscited by 3
- 2026Human-AI Synergy in Agentic Code Reviewcited by 3
- 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Timecited by 3
- 2026A Dataset of Agentic AI Coding Tool Configurationscited by 3
- 2026Agentic AI in Industry: Adoption Level and Deployment Barrierscited by 3
- 2026Why Are Agentic Pull Requests Merged or Rejected? An Empirical Studycited by 3
- 2026Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metricscited by 2
- 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requestscited by 2
- 2026Do AI Coding Agents Log Like Humans? An Empirical Studycited by 2
- 2026To What Extent Does Agent-generated Code Require Maintenance? An Empirical Studycited by 2
- 2026Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Developmentcited by 2
- 2026The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behaviorcited by 2
- 2025User Misconceptions of LLM-Based Conversational Programming Assistantscited by 1
- 2025GLOW: Graph-Language Co-Encoding for Agentic Workflow Performance Predictioncited by 1
- 2025How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requestscited by 1
- 2026On the Adoption of AI Coding Agents in Open-source Android and iOS Developmentcited by 1
- 2026On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositoriescited by 1
- 2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Reviewcited by 1
- 2026A Taxonomy of Runtime Faults in Model Context Protocol Serverscited by 1
- 2026All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Codecited by 1
- 2026RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescuecited by 1
- 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Softwarecited by 1
- 2026Do AI Agents Really Improve Code Readability?cited by 0
- 2026Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraphcited by 0
- 2026Nice Fold or Hero Call: Learning Budget-Efficient Thinking for Adaptive Reasoningcited by 0
- 2026Quality and Security Signals in AI-Generated Python Refactoring Pull Requestscited by 0
- 2026Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Datasetcited by 0
- 2026Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoptioncited by 0
- 2026Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Codecited by 0
- 2026From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Qualitycited by 0
- 2026Early Adoption of Agentic Coding Tools by GitHub Projectscited by 0
- 2026Test Coverage Analysis of Agentic Pull Requestscited by 0
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requestscited by 0
- 2026A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communitiescited by 0
- 2026On the Maintenance and Co-evolution of Agent Plugins: An Empirical Study of Claude Code Plugin Marketplacescited by 0
- 2026What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Revealcited by 0
50 most cited of 53 All 53 (JSON)
- 2026CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?29 Sep 2026
- 2026Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows23 Sep 2026
- 2026A Lean and Spec-Driven AI-Assisted Software Development Lifecycle for Applied AI Education: The AI-SDLC Approach21 Sep 2026
- 2026What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal1 Sep 2026
- 2026On the Maintenance and Co-evolution of Agent Plugins: An Empirical Study of Claude Code Plugin Marketplaces28 Aug 2026
- 2026The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior11 Aug 2026
- 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software5 Aug 2026
- 2026A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities29 Jul 2026
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests23 Jul 2026
- 2026Test Coverage Analysis of Agentic Pull Requests20 Jul 2026
- 2026Early Adoption of Agentic Coding Tools by GitHub Projects15 Jul 2026
- 2026From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality14 Jul 2026
38 more
- 2026Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code10 Jul 2026
- 2026RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescue1 Jul 2026
- 2026Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoption24 Jun 2026
- 2026All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code16 Jun 2026
- 2026Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset11 Jun 2026
- 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents3 Jun 2026
- 2026A Taxonomy of Runtime Faults in Model Context Protocol Servers3 Jun 2026
- 2026Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study21 May 2026
- 2026Quality and Security Signals in AI-Generated Python Refactoring Pull Requests20 May 2026
- 2026Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development19 May 2026
- 2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review17 May 2026
- 2026Agentic AI in Industry: Adoption Level and Deployment Barriers14 May 2026
- 2026Nice Fold or Hero Call: Learning Budget-Efficient Thinking for Adaptive Reasoning12 May 2026
- 2026A Dataset of Agentic AI Coding Tool Configurations8 May 2026
- 2026To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study7 May 2026
- 2026On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories27 Apr 2026
- 2026Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraph20 Apr 2026
- 2026Do AI Coding Agents Log Like Humans? An Empirical Study10 Apr 2026
- 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time1 Apr 2026
- 2026Human-AI Synergy in Agentic Code Review16 Mar 2026
- 2026Do AI Agents Really Improve Code Readability?14 Mar 2026
- 2026What to Cut? Predicting Unnecessary Methods in Agentic Code Generation19 Feb 2026
- 2026On the Adoption of AI Coding Agents in Open-source Android and iOS Development12 Feb 2026
- 2026Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents4 Feb 2026
- 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study29 Jan 2026
- 2026Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents27 Jan 2026
- 2026Understanding Dominant Themes in Reviewing Agentic AI-authored Code27 Jan 2026
- 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests24 Jan 2026
- 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests24 Jan 2026
- 2026AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development20 Jan 2026
- 2026Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests8 Jan 2026
- 2026Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics5 Jan 2026
- 2026Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub1 Jan 2026
- 2025How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requests31 Dec 2025
- 2025How Do Agents Perform Code Optimization? An Empirical Study25 Dec 2025
- 2025GLOW: Graph-Language Co-Encoding for Agentic Workflow Performance Prediction11 Dec 2025
- 2025Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning4 Dec 2025
- 2025Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projects12 Nov 2025
50 newest of 53 All 53 (JSON)
- 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
- 2025Agentic Refactoring: An Empirical Study of AI Coding Agentscited by 17
- 2025Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projectscited by 13
- 2026AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Developmentcited by 10
- 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requestscited by 7
- 2026Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHubcited by 6
- 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agentscited by 6
- 2026What to Cut? Predicting Unnecessary Methods in Agentic Code Generationcited by 5
- 2026Understanding Dominant Themes in Reviewing Agentic AI-authored Codecited by 4
- 2026Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requestscited by 3
- 2026Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agentscited by 3
- 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Studycited by 3
12 most cited of 47 All 47 (JSON)
- 2026Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows23 Sep 2026
- 2026A Lean and Spec-Driven AI-Assisted Software Development Lifecycle for Applied AI Education: The AI-SDLC Approach21 Sep 2026
- 2026What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal1 Sep 2026
- 2026On the Maintenance and Co-evolution of Agent Plugins: An Empirical Study of Claude Code Plugin Marketplaces28 Aug 2026
- 2026The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior11 Aug 2026
- 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software5 Aug 2026
- 2026A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities29 Jul 2026
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests23 Jul 2026
- 2026Test Coverage Analysis of Agentic Pull Requests20 Jul 2026
- 2026Early Adoption of Agentic Coding Tools by GitHub Projects15 Jul 2026
- 2026From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality14 Jul 2026
- 2026Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code10 Jul 2026
12 newest of 47 All 47 (JSON)
- 2025How Do Agents Perform Code Optimization? An Empirical Studycited by 4
- 2025User Misconceptions of LLM-Based Conversational Programming Assistantscited by 1
- 2026Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraphcited by 0
- 2026CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?cited by 0
- 2026CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?29 Sep 2026
- 2026Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraph20 Apr 2026
- 2025How Do Agents Perform Code Optimization? An Empirical Study25 Dec 2025
- 2025User Misconceptions of LLM-Based Conversational Programming Assistants29 Oct 2025
Similar, not cited
Close in wording, no citation either way
- 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
- 2026Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
- 2026On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source
- 2026Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild
- 2026Humans Integrate, Agents Fix: How Agent-Authored Pull Requests Are Referenced in Practice
- 2025A Study of Library Usage in Agent-Authored Pull Requests
- 2026Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests
- 2026A Task-Level Evaluation of AI Agents in Open-Source Projects