# Investigating The Smells of LLM Generated Code

arXiv: https://arxiv.org/abs/2510.03029 · HTML: https://mlsanity.com/paper/2510.03029/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 62 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code](https://mlsanity.com/paper/2404.00971/) (2024)
- [Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues](https://mlsanity.com/paper/2307.12596/) (2023)
- [Automated Repair of Programs from Large Language Models](https://mlsanity.com/paper/2205.10583/) (2022)
- [Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review](https://mlsanity.com/paper/2406.12655/) (2024)
- [Code Smells and Refactoring: A Tertiary Systematic Review of Challenges and Observations](https://mlsanity.com/paper/2004.10777/) (2020)
- [How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study](https://mlsanity.com/paper/2412.18989/) (2024)
- [User Centric Evaluation of Code Generation Tools](https://mlsanity.com/paper/2402.03130/) (2024)
- [Clean Code, Better Models: Enhancing LLM Performance with Smell-Cleaned Dataset](https://mlsanity.com/paper/2508.11958/) (2025)
- [ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation](https://mlsanity.com/paper/2406.12635/) (2024)

## Built on by (most cited first, 9 of 9)

- [Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions](https://mlsanity.com/paper/2602.14878/) (2026)
- [What to Cut? Predicting Unnecessary Methods in Agentic Code Generation](https://mlsanity.com/paper/2602.17091/) (2026)
- [What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants](https://mlsanity.com/paper/2605.30777/) (2026)
- [To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study](https://mlsanity.com/paper/2605.06464/) (2026)
- [Is Agent Code Less Maintainable Than Human Code?](https://mlsanity.com/paper/2606.21804/) (2026)
- [Can AI Agents Generate Microservices? How Far are We?](https://mlsanity.com/paper/2603.09004/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code](https://mlsanity.com/paper/2607.09902/) (2026)
- [What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code](https://mlsanity.com/paper/2609.12708/) (2026)

## Abstract (arXiv)

Context: Large Language Models (LLMs) are increasingly being used to generate program code. Much research has been reported on the functional correctness of generated code, but there is far less on code quality. Objectives: In this study, we propose a scenario-based method of evaluating the quality of LLM-generated code to identify the weakest scenarios, for which the quality of LLM-generated code should be improved. Methods: The method measures code smells, an important indicator of code quality, and compares them with a baseline formed from reference solutions of professionally written code. The test dataset is divided into various subsets according to the topics of the code and complexity of the coding tasks to represent different scenarios of using LLMs for code generation. We will also present an automated test system for this purpose and report experiments with the Java programs generated in response to prompts given to four state-of-the-art LLMs: Gemini Pro, ChatGPT, Codex, and Falcon. Results: We find that LLM-generated code has a higher incidence of code smells compared to reference solutions. Falcon performed the least badly, with a smell increase of 42.28%, followed by Gemini Pro (62.07%), ChatGPT (65.05%) and finally Codex (84.97%). The average smell increase across all LLMs was 63.34%, comprising 73.35% for implementation smells and 21.42% for design smells. We also found that the increase in code smells is greater for more complex coding tasks and for more advanced topics, such as those involving object-orientated concepts. Conclusion: In terms of code smells, LLMs' performances on various coding task complexities and topics are highly correlated to the quality of human-written code in the corresponding scenarios. However, the quality of LLM-generated code is noticeably poorer than human-written code.