# Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3

arXiv: https://arxiv.org/abs/2504.16027 · HTML: https://mlsanity.com/paper/2504.16027/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [Analysis of ChatGPT on Source Code](https://mlsanity.com/paper/2306.00597/) (2023)
- [Token-Hungry, Yet Precise: DeepSeek R1 Highlights the Need for Multi-Step Reasoning Over Speed in MATH](https://mlsanity.com/paper/2501.18576/) (2025)
- [Coding by Design: GPT-4 empowers Agile Model Driven Development](https://mlsanity.com/paper/2310.04304/) (2023)
- [Evaluating Large Language Models in Detecting Test Smells](https://mlsanity.com/paper/2407.19261/) (2024)
- [How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study](https://mlsanity.com/paper/2412.18989/) (2024)
- [ChatGPT as a Software Development Bot: A Project-based Study](https://mlsanity.com/paper/2310.13648/) (2023)
- [Are SonarQube Rules Inducing Bugs?](https://mlsanity.com/paper/1907.00376/) (2019)
- [Code Smells in Machine Learning Systems](https://mlsanity.com/paper/2203.00803/) (2022)
- [A Catalog of Transformations to Remove Smells From Natural Language Tests](https://mlsanity.com/paper/2404.16992/) (2024)
- [Code smells](https://mlsanity.com/paper/1802.06063/) (2018)

## Built on by (most cited first, 5 of 5)

- [Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions](https://mlsanity.com/paper/2602.14878/) (2026)
- [Are We SOLID Yet? An Empirical Study on Prompting LLMs to Detect Design Principle Violations](https://mlsanity.com/paper/2509.03093/) (2025)
- [Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2602.06475/) (2026)
- [Conversational AI for Rapid Scientific Prototyping: A Case Study on ESA's ELOPE Competition](https://mlsanity.com/paper/2601.04920/) (2026)
- [DynamicsLLM: a Dynamic Analysis-based Tool for Generating Intelligent Execution Traces Using LLMs to Detect Android Behavioural Code Smells](https://mlsanity.com/paper/2604.10661/) (2026)

## Abstract (arXiv)

Determining the most effective Large Language Model for code smell detection presents a complex challenge. This study introduces a structured methodology and evaluation matrix to tackle this issue, leveraging a curated dataset of code samples consistently annotated with known smells. The dataset spans four prominent programming languages Java, Python, JavaScript, and C++; allowing for cross language comparison. We benchmark two state of the art LLMs, OpenAI GPT 4.0 and DeepSeek-V3, using precision, recall, and F1 score as evaluation metrics. Our analysis covers three levels of detail: overall performance, category level performance, and individual code smell type performance. Additionally, we explore cost effectiveness by comparing the token based detection approach of GPT 4.0 with the pattern-matching techniques employed by DeepSeek V3. The study also includes a cost analysis relative to traditional static analysis tools such as SonarQube. The findings offer valuable guidance for practitioners in selecting an efficient, cost effective solution for automated code smell detection