# How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study

arXiv: https://arxiv.org/abs/2412.18989 · HTML: https://mlsanity.com/paper/2412.18989/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 25 Dec 2024.

## Builds on (most cited first, 25 of 25)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- [CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models](https://mlsanity.com/paper/2302.04012/) (2023)
- [On Learning Meaningful Assert Statements for Unit Test Cases](https://mlsanity.com/paper/2002.05800/) (2020)
- [SALLM: Security Assessment of Generated Code](https://mlsanity.com/paper/2311.00889/) (2023)
- [A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research](https://mlsanity.com/paper/2009.06520/) (2020)
- [An Empirical Study on the Usage of Transformer Models for Code Completion](https://mlsanity.com/paper/2108.01585/) (2021)
- [Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs](https://mlsanity.com/paper/2406.12513/) (2024)
- [Benchmarking Causal Study to Interpret Large Language Models for Source Code](https://mlsanity.com/paper/2308.12415/) (2023)
- [DLAP: A Deep Learning Augmented Large Language Model Prompting Framework for Software Vulnerability Detection](https://mlsanity.com/paper/2405.01202/) (2024)
- [Toward a Theory of Causation for Interpreting Neural Code Models](https://mlsanity.com/paper/2302.03788/) (2023)
- [Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations](https://mlsanity.com/paper/2407.08983/) (2024)
- [The Technical Debt Dataset](https://mlsanity.com/paper/1908.00827/) (2019)
- [Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?](https://mlsanity.com/paper/2401.01512/) (2024)
- [Evaluating Large Language Models in Detecting Test Smells](https://mlsanity.com/paper/2407.19261/) (2024)
- [A systematic literature review on the code smells datasets and validation mechanisms](https://mlsanity.com/paper/2306.01377/) (2023)
- [Prompt Learning for Multi-Label Code Smell Detection: A Promising Approach](https://mlsanity.com/paper/2402.10398/) (2024)
- [Vulnerability Handling of AI-Generated Code -- Existing Solutions and Open Challenges](https://mlsanity.com/paper/2408.08549/) (2024)
- [DACOS-A Manually Annotated Dataset of Code Smells](https://mlsanity.com/paper/2303.08729/) (2023)

## Built on by (most cited first, 6 of 6)

- [Investigating The Smells of LLM Generated Code](https://mlsanity.com/paper/2510.03029/) (2025)
- [Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3](https://mlsanity.com/paper/2504.16027/) (2025)
- [Optimizing Token Consumption in LLMs: A Nano Surge Approach for Code Reasoning Efficiency](https://mlsanity.com/paper/2504.15989/) (2025)
- [Secure Code Generation at Scale with Reflexion](https://mlsanity.com/paper/2511.03898/) (2025)
- [TOSSS: a CVE-based Software Security Benchmark for Large Language Models](https://mlsanity.com/paper/2603.10969/) (2026)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown significant potential in automating software engineering tasks, particularly in code generation. However, current evaluation benchmarks, which primarily focus on accuracy, fall short in assessing the quality of the code generated by these models, specifically their tendency to produce code smells. To address this limitation, we introduce CodeSmellEval, a benchmark designed to evaluate the propensity of LLMs for generating code smells. Our benchmark includes a novel metric: Propensity Smelly Score (PSC), and a curated dataset of method-level code smells: CodeSmellData. To demonstrate the use of CodeSmellEval, we conducted a case study with two state-of-the-art LLMs, CodeLlama and Mistral. The results reveal that both models tend to generate code smells, such as simplifiable-condition and consider-merging-isinstance. These findings highlight the effectiveness of our benchmark in evaluating LLMs, providing valuable insights into their reliability and their propensity to introduce code smells in code generation tasks.