mlsanityData to 7 Oct 2026

How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study

arXiv 2412.18989 ↗First posted 25 Dec 2024cs.SE cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown significant potential in automating software engineering tasks, particularly in code generation. However, current evaluation benchmarks, which primarily focus on accuracy, fall short in assessing the quality of the code generated by these models, specifically their tendency to produce code smells. To address this limitation, we introduce CodeSmellEval, a benchmark designed to evaluate the propensity of LLMs for generating code smells. Our benchmark includes a novel metric: Propensity Smelly Score (PSC), and a curated dataset of method-level code smells: CodeSmellData. To demonstrate the use of CodeSmellEval, we conducted a case study with two state-of-the-art LLMs, CodeLlama and Mistral. The results reveal that both models tend to generate code smells, such as simplifiable-condition and consider-merging-isinstance. These findings highlight the effectiveness of our benchmark in evaluating LLMs, providing valuable insights into their reliability and their propensity to introduce code smells in code generation tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 25 Dec 2024.

Builds on

25 of 43 references.

References · 25

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2023Mistral 7Bcited by 3,065
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  6. 2015Visualizing and Understanding Recurrent Networkscited by 158
  7. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  8. 2023InferFix: End-to-End Program Repair with LLMscited by 113
  9. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 52
  10. 2020On Learning Meaningful Assert Statements for Unit Test Casescited by 42
  11. 2020A Systematic Literature Review on the Use of Deep Learning in Software Engineering Researchcited by 35
  12. 2023SALLM: Security Assessment of Generated Codecited by 35
13 more
  1. 2021An Empirical Study on the Usage of Transformer Models for Code Completioncited by 22
  2. 2024Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMscited by 17
  3. 2023Benchmarking Causal Study to Interpret Large Language Models for Source Codecited by 14
  4. 2024DLAP: A Deep Learning Augmented Large Language Model Prompting Framework for Software Vulnerability Detectioncited by 13
  5. 2023Toward a Theory of Causation for Interpreting Neural Code Modelscited by 12
  6. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanationscited by 10
  7. 2019The Technical Debt Datasetcited by 6
  8. 2024Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?cited by 6
  9. 2024Evaluating Large Language Models in Detecting Test Smellscited by 6
  10. 2024Prompt Learning for Multi-Label Code Smell Detection: A Promising Approachcited by 3
  11. 2024Vulnerability Handling of AI-Generated Code -- Existing Solutions and Open Challengescited by 3
  12. 2023DACOS-A Manually Annotated Dataset of Code Smellscited by 2
  13. 2023A systematic literature review on the code smells datasets and validation mechanismscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Investigating The Smells of LLM Generated Codecited by 9
  2. 2025Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3cited by 5
  3. 2025Optimizing Token Consumption in LLMs: A Nano Surge Approach for Code Reasoning Efficiencycited by 2
  4. 2025Secure Code Generation at Scale with Reflexioncited by 1
  5. 2026TOSSS: a CVE-based Software Security Benchmark for Large Language Modelscited by 1
  6. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1