mlsanityData to 14 Oct 2026

Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild

arXiv 2609.17598 ↗First posted 12 Sep 2026cs.SEcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Autonomous coding agents now open pull requests in public repositories at a scale that was out of reach two years ago, yet little is known about what happens to that code after it lands. This paper studies 37,623 provenance-labeled pull requests (PRs) from five commercial agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code) and a matched human baseline, drawn from 2,807 GitHub repositories between December 2024 and July 2025. We combine the AIDev dataset with 58,792 cached GitHub API responses to measure security smells in added code, structural maintainability, post-merge churn, revert rates, and human review behavior. Three results stand out. First, quality differences are vendor-specific rather than uniform: Codex-authored PRs were reverted about half as often as human PRs (6.1% vs. 11.5%, odds ratio 0.50), while Devin PRs were reverted more often (14.5%, odds ratio 1.31). Second, agent code pooled across vendors was less likely than human code to contain a security smell (odds ratio 0.63), driven by fewer hardcoded credentials and eval-style constructs. Third, review effort concentrates unevenly: Copilot PRs drew the most human reviews and change requests, and Claude Code PRs waited the longest for a first human review (median 12.6 hours). All pipeline code, statistical reports, and figures are released for replication.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 19 references that are in the map. First posted on arXiv 12 Sep 2026.

Builds on

10 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 7

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  4. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  5. 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
  6. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 91
  7. 2023Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Studycited by 38

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
  2. 2026Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests
  3. 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
  4. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
  5. 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
  6. 2026Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
  7. 2026Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents
  8. 2026Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub