# What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects

arXiv: https://arxiv.org/abs/2509.07763 · HTML: https://mlsanity.com/paper/2509.07763/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 84 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Builds on (most cited first, 22 of 22)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Review of Generalized Zero-Shot Learning Methods](https://mlsanity.com/paper/2011.08641/) (2020)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward](https://mlsanity.com/paper/2406.07765/) (2024)
- [Refactoring Programs Using Large Language Models with Few-Shot Examples](https://mlsanity.com/paper/2311.11690/) (2023)
- [Code Smells and Refactoring: A Tertiary Systematic Review of Challenges and Observations](https://mlsanity.com/paper/2004.10777/) (2020)
- [The impact of feature importance methods on the interpretation of defect classifiers](https://mlsanity.com/paper/2202.02389/) (2022)
- [Why We Refactor? Confessions of GitHub Contributors](https://mlsanity.com/paper/1607.02459/) (2016)
- [An Empirical Study on the Potential of LLMs in Automated Software Refactoring](https://mlsanity.com/paper/2411.04444/) (2024)
- [A Critical Comparison on Six Static Analysis Tools: Detection, Agreement, and Precision](https://mlsanity.com/paper/2101.08832/) (2021)
- [Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis](https://mlsanity.com/paper/2406.10273/) (2024)
- [On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?](https://mlsanity.com/paper/2412.11698/) (2024)
- [EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs](https://mlsanity.com/paper/2405.20551/) (2024)
- [An Insight into the Pull Requests of GitHub](https://mlsanity.com/paper/1807.01853/) (2018)
- [Recommendation of Move Method Refactoring Using Path-Based Representation of Code](https://mlsanity.com/paper/2002.06392/) (2020)
- [Evaluating Time-Dependent Methods and Seasonal Effects in Code Technical Debt Prediction](https://mlsanity.com/paper/2408.08095/) (2024)
- [A Novel Refactoring and Semantic Aware Abstract Syntax Tree Differencing Tool and a Benchmark for Evaluating the Accuracy of Diff Tools](https://mlsanity.com/paper/2403.05939/) (2024)
- [A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering](https://mlsanity.com/paper/2501.12728/) (2025)
- [Why Developers Refactor Source Code: A Mining-based Study](https://mlsanity.com/paper/2101.01430/) (2021)
- [How Do I Refactor This? An Empirical Study on Refactoring Trends and Topics in Stack Overflow](https://mlsanity.com/paper/2110.12229/) (2021)

## Built on by (most cited first, 4 of 4)

- [A Defect is Being Born: How Close Are We? A Time Sensitive Forecasting Approach](https://mlsanity.com/paper/2601.01921/) (2026)
- [Evaluating Large Language Models for Detecting Architectural Decision Violations](https://mlsanity.com/paper/2602.07609/) (2026)
- [Making Sense of AI Agents Hype: Adoption, Architectures, and Takeaways from Practitioners](https://mlsanity.com/paper/2604.00189/) (2026)
- [AI Writes Code, Humans Pay the Debt. An Empirical Study on the Sustainability and Evolution of Agent-Generated Code](https://mlsanity.com/paper/2609.04208/) (2026)

## Abstract (arXiv)

Context. Code refactoring improves software quality without changing external behavior. Despite its advantages, its benefits are hindered by the considerable cost of time, resources, and continuous effort it demands. Aim. Understanding why developers refactor, and which metrics capture these motivations, may support wider and more effective use of refactoring in practice. Method. We performed a large-scale empirical study to analyze developers refactoring activity, leveraging Large Language Models (LLMs) to identify underlying motivations from version control data, comparing our findings with previous motivations reported in the literature. Results. LLMs matched human judgment in 80% of cases, but aligned with literature-based motivations in only 47%. They enriched 22% of motivations with more detailed rationale, often highlighting readability, clarity, and structural improvements. Most motivations were pragmatic, focused on simplification and maintainability. While metrics related to developer experience and code readability ranked highest, their correlation with motivation categories was weak. Conclusions. We conclude that LLMs effectively capture surface-level motivations but struggle with architectural reasoning. Their value lies in providing localized explanations, which, when combined with software metrics, can form hybrid approaches. Such integration offers a promising path toward prioritizing refactoring more systematically and balancing short-term improvements with long-term architectural goals.