mlsanityData to 14 Oct 2026

Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects

arXiv 2502.02368 ↗First posted 4 Feb 2025cs.SE cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have gained attention for addressing coding problems, but their effectiveness in fixing code maintainability remains unclear. This study evaluates LLMs capability to resolve 127 maintainability issues from 10 GitHub repositories. We use zero-shot prompting for Copilot Chat and Llama 3.1, and few-shot prompting with Llama only. The LLM-generated solutions are assessed for compilation errors, test failures, and new maintainability problems. Llama with few-shot prompting successfully fixed 44.9% of the methods, while Copilot Chat and Llama zero-shot fixed 32.29% and 30%, respectively. However, most solutions introduced errors or new maintainability issues. We also conducted a human study with 45 participants to evaluate the readability of 51 LLM-generated solutions. The human study showed that 68.63% of participants observed improved readability. Overall, while LLMs show potential for fixing maintainability issues, their introduction of errors highlights their current limitations.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 22 of its 51 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

22 of 51 references.

References · 22

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 463
  3. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  4. 2023Using an LLM to Help With Code Understandingcited by 228
  5. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  6. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  7. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 67
  8. 2023Large Language Models for Test-Free Fault Localizationcited by 67
  9. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  10. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  11. 2023What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?cited by 47
  12. 2023No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPTcited by 47
10 more
  1. 2024AgentFL: Scaling LLM-based Fault Localization to Project-Level Contextcited by 47
  2. 2022Automatic Code Documentation Generation Using GPT-3cited by 37
  3. 2024LLMParser: An Exploratory Study on Using Large Language Models for Log Parsingcited by 32
  4. 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 29
  5. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 28
  6. 2022How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilotcited by 14
  7. 2024Software Vulnerability and Functionality Assessment using LLMscited by 11
  8. 2024CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarioscited by 11
  9. 2024EM-Assist: Safe Automated ExtractMethod Refactoring with LLMscited by 8
  10. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025textttDroid: A Resource Suite for AI-Generated Code Detectioncited by 9
  2. 2026To What Extent Does Agent-generated Code Require Maintenance? An Empirical Studycited by 2
  3. 2025A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standardscited by 1
  4. 2026Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Editscited by 1