# SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

arXiv: https://arxiv.org/abs/2507.12415 · HTML: https://mlsanity.com/paper/2507.12415/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2025.

## Built on by (most cited first, 30 of 36)

- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?](https://mlsanity.com/paper/2511.06090/) (2025)
- [Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?](https://mlsanity.com/paper/2602.11988/) (2026)
- [SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints](https://mlsanity.com/paper/2509.09853/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Training Versatile Coding Agents in Synthetic Environments](https://mlsanity.com/paper/2512.12216/) (2025)
- [SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks](https://mlsanity.com/paper/2607.00053/) (2026)
- [SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution](https://mlsanity.com/paper/2603.13428/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?](https://mlsanity.com/paper/2602.19594/) (2026)
- [FormulaCode: Evaluating Agentic Optimization on Large Codebases](https://mlsanity.com/paper/2603.16011/) (2026)
- [SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution](https://mlsanity.com/paper/2605.08366/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [Evolving Excellence: Automated Optimization of LLM-based Agents](https://mlsanity.com/paper/2512.09108/) (2025)
- [VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?](https://mlsanity.com/paper/2605.06068/) (2026)
- [SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering](https://mlsanity.com/paper/2605.17526/) (2026)
- [GA4GC: Greener Agent for Greener Code via Multi-Objective Configuration Optimization](https://mlsanity.com/paper/2510.04135/) (2025)
- [PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization](https://mlsanity.com/paper/2607.19653/) (2026)
- [When Models Edit Too Much: On the Fidelity of Minimal Code Edits](https://mlsanity.com/paper/2609.04061/) (2026)
- [Evaluating LLMs on Real-World Software Performance Optimization](https://mlsanity.com/paper/2606.25530/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [Lean Refactor: Multi-Objective Controllable Proof Optimization via Agentic Strategy Search](https://mlsanity.com/paper/2605.20244/) (2026)
- [Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?](https://mlsanity.com/paper/2607.01211/) (2026)
- [PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks](https://mlsanity.com/paper/2607.28587/) (2026)
- [SemRep: Generative Code Representation Learning with Code Transformations](https://mlsanity.com/paper/2603.13640/) (2026)
- [EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback](https://mlsanity.com/paper/2604.05137/) (2026)
- [BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledge](https://mlsanity.com/paper/2605.15815/) (2026)
- [Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproduction](https://mlsanity.com/paper/2607.01764/) (2026)
- [Rethinking Code Performance Benchmarks for LLMs](https://mlsanity.com/paper/2607.07619/) (2026)

## Abstract (arXiv)

Code performance optimization is paramount in real-world software engineering and critical for production-level systems. While Large Language Models (LLMs) have demonstrated impressive capabilities in code generation and bug fixing, their proficiency in enhancing code performance at the repository level remains largely unexplored. To address this gap, we introduce SWE-Perf, the first benchmark specifically designed to systematically evaluate LLMs on code performance optimization tasks within authentic repository contexts. SWE-Perf comprises 140 carefully curated instances, each derived from performance-improving pull requests from popular GitHub repositories. Each benchmark instance includes the relevant codebase, target functions, performance-related tests, expert-authored patches, and executable environments. Through a comprehensive evaluation of representative methods that span file-level and repo-level approaches (e.g., Agentless and OpenHands), we reveal a substantial capability gap between existing LLMs and expert-level optimization performance, highlighting critical research opportunities in this emerging field.