# Quality and Security Signals in AI-Generated Python Refactoring Pull Requests

arXiv: https://arxiv.org/abs/2605.21453 · HTML: https://mlsanity.com/paper/2605.21453/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 14 of 14)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub](https://mlsanity.com/paper/2509.14745/) (2025)
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study](https://mlsanity.com/paper/2310.02059/) (2023)
- [Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap](https://mlsanity.com/paper/2410.06107/) (2024)
- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [AIDev: Studying AI Coding Agents on GitHub](https://mlsanity.com/paper/2602.09185/) (2026)
- [The Prevalence of Code Smells in Machine Learning projects](https://mlsanity.com/paper/2103.04146/) (2021)
- [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://mlsanity.com/paper/2506.23034/) (2025)
- [Is LLM-Generated Code More Maintainable & Reliable than Human-Written Code?](https://mlsanity.com/paper/2508.00700/) (2025)
- [From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu](https://mlsanity.com/paper/2511.02827/) (2025)
- [AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality](https://mlsanity.com/paper/2601.16839/) (2026)

## Abstract (arXiv)

As AI agents increasingly contribute to code development and maintenance, there is still limited empirical evidence on the quality and risk characteristics of their changes in real-world projects, particularly for refactoring-oriented contributions. It remains unclear how agent-authored refactoring edits affect maintainability, code quality, and security once merged into GitHub repositories. To address this gap, we conduct an empirical study of Python refactoring pull requests (PRs) from the AIDev dataset. We analyze agentic refactoring PRs using PyQu, an ML-based quality assessment tool for Python, to quantify changes across five quality attributes, and we complement PyQu with domain-independent static analysis (Pylint and Bandit) to measure code quality and security issues before and after each change. Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes, with usability improving most frequently (36.5%). At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations such as long lines-while 4.7% introduce new Bandit findings. From the observed diffs, we derive a taxonomy of 24 recurring change operations and map them to the lint and security findings they most commonly affect. Despite these mixed outcomes, developer acceptance is high: 73.5% of the analyzed PRs are merged, including cases that introduce new lint or security findings, often alongside the removal of existing issues. Overall, these findings highlight both the promise and current limitations of agentic refactoring, and motivate stronger tool-in-the-loop quality and security gating for AI-driven development workflows.