mlsanityData to 14 Oct 2026

Quality and Security Signals in AI-Generated Python Refactoring Pull Requests

arXiv 2605.21453 ↗First posted 20 May 2026cs.SE cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

As AI agents increasingly contribute to code development and maintenance, there is still limited empirical evidence on the quality and risk characteristics of their changes in real-world projects, particularly for refactoring-oriented contributions. It remains unclear how agent-authored refactoring edits affect maintainability, code quality, and security once merged into GitHub repositories. To address this gap, we conduct an empirical study of Python refactoring pull requests (PRs) from the AIDev dataset. We analyze agentic refactoring PRs using PyQu, an ML-based quality assessment tool for Python, to quantify changes across five quality attributes, and we complement PyQu with domain-independent static analysis (Pylint and Bandit) to measure code quality and security issues before and after each change. Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes, with usability improving most frequently (36.5%). At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations such as long lines-while 4.7% introduce new Bandit findings. From the observed diffs, we derive a taxonomy of 24 recurring change operations and map them to the lint and security findings they most commonly affect. Despite these mixed outcomes, developer acceptance is high: 73.5% of the analyzed PRs are merged, including cases that introduce new lint or security findings, often alongside the removal of existing issues. Overall, these findings highlight both the promise and current limitations of agentic refactoring, and motivate stronger tool-in-the-loop quality and security gating for AI-driven development workflows.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 33 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

14 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026AIDev: Studying AI Coding Agents on GitHub9 Feb
  2. 2026AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality23 Jan
  3. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub18 Sep

Other references · 11

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  3. 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
  4. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  5. 2023Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Studycited by 38
  6. 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
  7. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
  8. 2021The Prevalence of Code Smells in Machine Learning projectscited by 16
  9. 2025Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generationcited by 13
  10. 2025Is LLM-Generated Code More Maintainable & Reliable than Human-Written Code?cited by 5
  11. 2025From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQucited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Do AI Agents Really Improve Code Readability?
  2. 2026Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests
  3. 2026Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
  4. 2026Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild
  5. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
  6. 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
  7. 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
  8. 2025Agentic Refactoring: An Empirical Study of AI Coding Agents