# LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

arXiv: https://arxiv.org/abs/2610.20027 · HTML: https://mlsanity.com/paper/2610.20027/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Builds on (most cited first, 20 of 20)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step](https://mlsanity.com/paper/2402.16906/) (2024)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)
- [CodeContests+: High-Quality Test Case Generation for Competitive Programming](https://mlsanity.com/paper/2506.05817/) (2025)
- [RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code](https://mlsanity.com/paper/2503.07832/) (2025)
- [NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness](https://mlsanity.com/paper/2401.15963/) (2024)
- [Training Language Models to Generate Quality Code with Program Analysis Feedback](https://mlsanity.com/paper/2505.22704/) (2025)
- [Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5](https://mlsanity.com/paper/2211.14875/) (2022)
- [FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks](https://mlsanity.com/paper/2504.06939/) (2025)
- [CoCoST: Automatic Complex Code Generation with Online Searching and Correctness Testing](https://mlsanity.com/paper/2403.13583/) (2024)
- [Improving LLM-Generated Code Quality with GRPO](https://mlsanity.com/paper/2506.02211/) (2025)
- [Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness](https://mlsanity.com/paper/2508.14419/) (2025)
- [The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks](https://mlsanity.com/paper/2402.17679/) (2024)
- [Ghost Echoes Revealed: Benchmarking Maintainability Metrics and Machine Learning Predictions Against Human Assessments](https://mlsanity.com/paper/2408.10754/) (2024)
- [A Large-Scale Collection Of (Non-)Actionable Static Code Analysis Reports](https://mlsanity.com/paper/2511.10323/) (2025)
- [Towards Extending the Range of Bugs That Automated Program Repair Can Handle](https://mlsanity.com/paper/2211.03911/) (2022)
- [BitsAI-Fix: LLM-Driven Approach for Automated Lint Error Resolution in Practice](https://mlsanity.com/paper/2508.03487/) (2025)

## Abstract (arXiv)

Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other patterns in source files. However, existing code benchmarks do not explicitly evaluate linter-guided file refactoring, where models must reduce lint alerts in source files with correct initial functionality while satisfying functionality preservation. We introduce LintEval, a multilingual benchmark for linter-guided file refactoring across C++, Python, and Java, comprising 13,061 programs with correct initial functionality and 171,662 lint alerts extracted using language-specific linters. LintEval jointly evaluates functionality preservation, lint alert reduction, new alert avoidance, and edit minimality. Experiments reveal a persistent trade-off: closed-source commercial models reach up to 90.4% Reduction@P but achieve only 67.1%-68.3% Pass@1, while several open-source small models show low or negative Reduction@P. A controlled fine-tuning study further shows that supervision format matters: refactoring-oriented SFT raises Qwen2.5-Coder-7B Reduction@P from 24.7% to 76.0%. LintEval enables controlled evaluation of models that improve source files under functionality preservation.