# An Empirical Study on the Code Refactoring Capability of Large Language Models

arXiv: https://arxiv.org/abs/2411.02320 · HTML: https://mlsanity.com/paper/2411.02320/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2024.

## Built on by (most cited first, 19 of 19)

- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring](https://mlsanity.com/paper/2511.03153/) (2025)
- [CodeTaste: Can LLMs Generate Human-Level Code Refactorings?](https://mlsanity.com/paper/2603.04177/) (2026)
- [SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring](https://mlsanity.com/paper/2608.09802/) (2026)
- [Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias](https://mlsanity.com/paper/2508.17361/) (2025)
- [SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair](https://mlsanity.com/paper/2605.07001/) (2026)
- [Optimizing Token Consumption in LLMs: A Nano Surge Approach for Code Reasoning Efficiency](https://mlsanity.com/paper/2504.15989/) (2025)
- [Taxonomy of migration scenarios for Qiskit refactoring using LLMs](https://mlsanity.com/paper/2506.07135/) (2025)
- [Can LLMs Replace Humans During Code Chunking?](https://mlsanity.com/paper/2506.19897/) (2025)
- [Refactoring with LLMs: Bridging Human Expertise and Machine Understanding](https://mlsanity.com/paper/2510.03914/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Automatic Qiskit Code Refactoring Using Large Language Models](https://mlsanity.com/paper/2506.14535/) (2025)
- [TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance](https://mlsanity.com/paper/2601.18241/) (2026)
- [AI-Assisted Unit Test Writing and Test-Driven Code Refactoring: A Case Study](https://mlsanity.com/paper/2604.03135/) (2026)
- [Structural Anchors and Reasoning Fragility:Understanding CoT Robustness in LLM4Code](https://mlsanity.com/paper/2604.12214/) (2026)
- [An Exploratory Case Study of LLM-Assisted Refactoring and Gameplay Feature Generation in an Endless Runner Game](https://mlsanity.com/paper/2606.21171/) (2026)
- [SDAD: Spec-Driven Agentic Development for the AI-Native SDLC](https://mlsanity.com/paper/2608.20341/) (2026)
- [REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring](https://mlsanity.com/paper/2608.23611/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown potential to enhance software development through automated code generation and refactoring, reducing development time and improving code quality. This study empirically evaluates StarCoder2, an LLM optimized for code generation, in refactoring code across 30 open-source Java projects. We compare StarCoder2's performance against human developers, focusing on (1) code quality improvements, (2) types and effectiveness of refactorings, and (3) enhancements through one-shot and chain-of-thought prompting. Our results indicate that StarCoder2 reduces code smells by 20.1% more than developers, excelling in systematic issues like Long Statement and Magic Number, while developers handle complex, context-dependent issues better. One-shot prompting increases the unit test pass rate by 6.15% and improves code smell reduction by 3.52%. Generating five refactorings per input further increases the pass rate by 28.8%, suggesting that combining one-shot prompting with multiple refactorings optimizes performance. These findings provide insights into StarCoder2's potential and best practices for integrating LLMs into software refactoring, supporting more efficient and effective code improvement in real-world applications.