# SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair

arXiv: https://arxiv.org/abs/2605.07001 · HTML: https://mlsanity.com/paper/2605.07001/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 27 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 6 of 6)

- [An Empirical Study on the Code Refactoring Capability of Large Language Models](https://mlsanity.com/paper/2411.02320/) (2024)
- [MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaboration](https://mlsanity.com/paper/2503.14340/) (2025)
- [RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring](https://mlsanity.com/paper/2511.03153/) (2025)
- [Clean Code, Better Models: Enhancing LLM Performance with Smell-Cleaned Dataset](https://mlsanity.com/paper/2508.11958/) (2025)
- [ROSE: Transformer-Based Refactoring Recommendation for Architectural Smells](https://mlsanity.com/paper/2507.12561/) (2025)
- [A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code](https://mlsanity.com/paper/2511.15817/) (2025)

## Built on by (most cited first, 3 of 3)

- [SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks](https://mlsanity.com/paper/2606.05574/) (2026)
- [REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring](https://mlsanity.com/paper/2608.23611/) (2026)
- [SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories](https://mlsanity.com/paper/2609.29465/) (2026)

## Abstract (arXiv)

Architectural code smells erode software maintainability and are costly to repair manually, yet unlike localized bugs, they require cross-module reasoning about design intent that challenges both developers and automated tools. While large language model agents excel at bug fixing and code-level refactoring, their ability to repair architectural code smells remains unexplored. We present the first empirical evaluation of LLM agents on architectural code smell repair. We contribute SmellBench, a task orchestration framework that incorporates smell-type-specific optimized prompts and supports iterative multi-step execution, together with a scoring methodology that separately evaluates repair effectiveness, false positive identification, and net codebase impact. We evaluate 11 agent configurations from four model families (GPT, Claude, Gemini, Mistral) on 65 hard-severity architectural smells detected by PyExamine in the Python project scikit-learn, validated against expert judgments. Expert validation reveals that 63.1% of detected smells are false positives, while the best agent achieves a 47.7% resolution rate. Agents identify false positives with up to kappa = 0.94 expert agreement, but repair aggressiveness and net codebase quality are inversely related: the most aggressive agent introduces 140 new smells. These findings expose a gap between current LLM capabilities in localized code transformations and the architectural understanding needed for cross-module refactoring. SmellBench provides reusable infrastructure for tracking progress on this underexplored dimension of automated software engineering. We release our code and data at https://doi.org/10.5281/zenodo.19247588.