# CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

arXiv: https://arxiv.org/abs/2603.04177 · HTML: https://mlsanity.com/paper/2603.04177/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 13 of 13)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation](https://mlsanity.com/paper/2503.06680/) (2025)
- [Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions](https://mlsanity.com/paper/2312.12450/) (2023)
- [RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code](https://mlsanity.com/paper/2503.07832/) (2025)
- [Refactoring Programs Using Large Language Models with Few-Shot Examples](https://mlsanity.com/paper/2311.11690/) (2023)
- [An Empirical Study on the Code Refactoring Capability of Large Language Models](https://mlsanity.com/paper/2411.02320/) (2024)
- [SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios](https://mlsanity.com/paper/2509.22097/) (2025)
- [SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring](https://mlsanity.com/paper/2602.03712/) (2026)
- [An Empirical Study on the Potential of LLMs in Automated Software Refactoring](https://mlsanity.com/paper/2411.04444/) (2024)
- [Refactoring Codebases through Library Design](https://mlsanity.com/paper/2506.11058/) (2025)

## Built on by (most cited first, 8 of 8)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring](https://mlsanity.com/paper/2608.09802/) (2026)
- [SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks](https://mlsanity.com/paper/2606.05574/) (2026)
- [Super Library Agent: Joint Generation and Maintenance of Multiple Applications Beyond the Single Codebase](https://mlsanity.com/paper/2608.29310/) (2026)
- [Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills](https://mlsanity.com/paper/2609.12012/) (2026)
- [Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models](https://mlsanity.com/paper/2609.35841/) (2026)
- [Can Agents Design Libraries for Agents?](https://mlsanity.com/paper/2609.36730/) (2026)
- [Chronos Enables Code Agents to Reason over Software Evolution](https://mlsanity.com/paper/2610.11578/) (2026)

## Abstract (arXiv)

LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. We investigate whether agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. To this end, we construct CodeTaste, a benchmark mined from large multi-file open-source refactorings. To score solutions, we combine repository test suites that measure functional correctness with tailored static checks that verify removal of undesired and introduction of desired code patterns using dataflow reasoning. Our results show a clear gap: agents perform well at implementing refactorings that are specified in detail, but often fail to discover the human refactoring choices when given a focus area for changes. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. We release the benchmark, leaderboard, and code.