# Refactoring Programs Using Large Language Models with Few-Shot Examples

arXiv: https://arxiv.org/abs/2311.11690 · HTML: https://mlsanity.com/paper/2311.11690/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [PanGu-Coder: Program Synthesis with Function-Level Language Modeling](https://mlsanity.com/paper/2207.11280/) (2022)
- [CoditT5: Pretraining for Source Code and Natural Language Editing](https://mlsanity.com/paper/2208.05446/) (2022)
- [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021)
- [CODIT: Code Editing with Tree-Based Neural Models](https://mlsanity.com/paper/1810.00314/) (2018)
- [Repairing Bugs in Python Assignments Using Large Language Models](https://mlsanity.com/paper/2209.14876/) (2022)
- [CodeEditor: Learning to Edit Source Code with Pre-trained Models](https://mlsanity.com/paper/2210.17040/) (2022)
- [ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback](https://mlsanity.com/paper/2107.14035/) (2021)
- [Program Repair with Minimal Edits Using CodeT5](https://mlsanity.com/paper/2309.14760/) (2023)
- [Improving Readability of Scratch Programs with Search-based Refactoring](https://mlsanity.com/paper/2108.07114/) (2021)

## Built on by (most cited first, 28 of 28)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement](https://mlsanity.com/paper/2411.00622/) (2024)
- [Agentic Refactoring: An Empirical Study of AI Coding Agents](https://mlsanity.com/paper/2511.04824/) (2025)
- [VerMCTS: Synthesizing Multi-Step Programs using a Verifier, a Large Language Model, and Tree Search](https://mlsanity.com/paper/2402.08147/) (2024)
- [What can Large Language Models Capture about Code Functional Equivalence?](https://mlsanity.com/paper/2408.11081/) (2024)
- [Migrating Code At Scale With LLMs At Google](https://mlsanity.com/paper/2504.09691/) (2025)
- [LLMs: A Game-Changer for Software Engineers?](https://mlsanity.com/paper/2411.00932/) (2024)
- [MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaboration](https://mlsanity.com/paper/2503.14340/) (2025)
- [CodeTaste: Can LLMs Generate Human-Level Code Refactorings?](https://mlsanity.com/paper/2603.04177/) (2026)
- [Are Coding Agents Generating Over-Mocked Tests? An Empirical Study](https://mlsanity.com/paper/2602.00409/) (2026)
- [Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects](https://mlsanity.com/paper/2502.02368/) (2025)
- [EditLord: Learning Code Transformation Rules for Code Editing](https://mlsanity.com/paper/2504.15284/) (2025)
- [What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects](https://mlsanity.com/paper/2509.07763/) (2025)
- [Skill-as-Pseudocode: Refactoring Skill Libraries to Pseudocode for LLM Agents](https://mlsanity.com/paper/2605.27955/) (2026)
- [AuPair: Golden Example Pairs for Code Repair](https://mlsanity.com/paper/2502.18487/) (2025)
- [LLM-based Multi-Agent System for Intelligent Refactoring of Haskell Code](https://mlsanity.com/paper/2506.19481/) (2025)
- [CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit](https://mlsanity.com/paper/2411.10842/) (2024)
- [Towards Secure Program Partitioning for Smart Contracts with LLM's In-Context Learning](https://mlsanity.com/paper/2502.14215/) (2025)
- [Generating High-Quality Datasets for Code Editing via Open-Source Language Models](https://mlsanity.com/paper/2509.25203/) (2025)
- [Refactoring with LLMs: Bridging Human Expertise and Machine Understanding](https://mlsanity.com/paper/2510.03914/) (2025)
- [Leveraging LLMs, IDEs, and Semantic Embeddings for Automated Move Method Refactoring](https://mlsanity.com/paper/2503.20934/) (2025)
- [Exploring the Security Threats of Retriever Backdoors in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2512.21681/) (2025)
- [TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks](https://mlsanity.com/paper/2601.22597/) (2026)
- [RefactorAssist: Agentic Refinement for Reliable Code Refactoring](https://mlsanity.com/paper/2608.00924/) (2026)
- [Automated Harmfulness Testing for Code Large Language Models](https://mlsanity.com/paper/2503.16740/) (2025)
- [Do AI Models Dream of Faster Code? An Empirical Study on LLM-Proposed Performance Improvements in Real-World Software](https://mlsanity.com/paper/2510.15494/) (2025)
- [AI-Assisted Unit Test Writing and Test-Driven Code Refactoring: A Case Study](https://mlsanity.com/paper/2604.03135/) (2026)
- [An Exploratory Case Study of LLM-Assisted Refactoring and Gameplay Feature Generation in an Endless Runner Game](https://mlsanity.com/paper/2606.21171/) (2026)

## Abstract (arXiv)

A less complex and more straightforward program is a crucial factor that enhances its maintainability and makes writing secure and bug-free programs easier. However, due to its heavy workload and the risks of breaking the working programs, programmers are reluctant to do code refactoring, and thus, it also causes the loss of potential learning experiences. To mitigate this, we demonstrate the application of using a large language model (LLM), GPT-3.5, to suggest less complex versions of the user-written Python program, aiming to encourage users to learn how to write better programs. We propose a method to leverage the prompting with few-shot examples of the LLM by selecting the best-suited code refactoring examples for each target programming problem based on the prior evaluation of prompting with the one-shot example. The quantitative evaluation shows that 95.68% of programs can be refactored by generating 10 candidates each, resulting in a 17.35% reduction in the average cyclomatic complexity and a 25.84% decrease in the average number of lines after filtering only generated programs that are semantically correct. Furthermore, the qualitative evaluation shows outstanding capability in code formatting, while unnecessary behaviors such as deleting or translating comments are also observed.