# SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks

arXiv: https://arxiv.org/abs/2606.05574 · HTML: https://mlsanity.com/paper/2606.05574/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 18 of 18)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://mlsanity.com/paper/2310.11248/) (2023)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code](https://mlsanity.com/paper/2503.07832/) (2025)
- [CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation](https://mlsanity.com/paper/2502.19166/) (2025)
- [SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring](https://mlsanity.com/paper/2602.03712/) (2026)
- [An Empirical Study on the Potential of LLMs in Automated Software Refactoring](https://mlsanity.com/paper/2411.04444/) (2024)
- [CodeTaste: Can LLMs Generate Human-Level Code Refactorings?](https://mlsanity.com/paper/2603.04177/) (2026)
- [PerfBench: Can Agents Resolve Real-World Performance Bugs?](https://mlsanity.com/paper/2509.24091/) (2025)
- [Code Refactoring with LLM: A Comprehensive Evaluation With Few-Shot Settings](https://mlsanity.com/paper/2511.21788/) (2025)
- [SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair](https://mlsanity.com/paper/2605.07001/) (2026)
- [Refactoring with LLMs: Bridging Human Expertise and Machine Understanding](https://mlsanity.com/paper/2510.03914/) (2025)

## Abstract (arXiv)

Code Agents have achieved remarkable advances in recent years, exhibiting strong capabilities across a wide range of software engineering tasks. However, their misuse often produces bloated and disorganized code that impairing readability, extensibility, and robustness. Despite this risk, existing benchmarks largely evaluate functional correctness rather than long-term maintainability of code agents. In this paper, we propose SmellBench, an extensible code refactoring benchmark that proactively injects code smells into clean code snippets from real-world repositories. This design enables the generation of controlled, high-quality, and diverse refactoring cases with human-written ground truth. Specifically, it contains 294 cases spanning 7 popular smell types, 3 difficulty levels, 2 instruction settings across 7 real-world repositories. We further design 3 evaluation aspects covering functional correctness, localization ability, and refactoring quality assessment. Experiments with 2 popular agents and 6 large langauge models (LLMs) show that the best combination - Qwen Code + Claude Sonnet 4.5 - achieved only a 50.34 score of smell elimination. Further analysis reveals that this gap arises from a focus on local code smells and a lack of cross-file understanding, which hinders comprehensive smell elimination.