mlsanityData to 7 Oct 2026

SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

arXiv 2507.12415 ↗First posted 16 Jul 2025cs.SEcited by 33

Not on the map. Why?

Abstract (arXiv)

Code performance optimization is paramount in real-world software engineering and critical for production-level systems. While Large Language Models (LLMs) have demonstrated impressive capabilities in code generation and bug fixing, their proficiency in enhancing code performance at the repository level remains largely unexplored. To address this gap, we introduce SWE-Perf, the first benchmark specifically designed to systematically evaluate LLMs on code performance optimization tasks within authentic repository contexts. SWE-Perf comprises 140 carefully curated instances, each derived from performance-improving pull requests from popular GitHub repositories. Each benchmark instance includes the relevant codebase, target functions, performance-related tests, expert-authored patches, and executable environments. Through a comprehensive evaluation of representative methods that span file-level and repo-level approaches (e.g., Agentless and OpenHands), we reveal a substantial capability gap between existing LLMs and expert-level optimization performance, highlighting critical research opportunities in this emerging field.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jul 2025.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 25.

33 papers.

Sort:

By area:

  1. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  2. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
  3. 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?cited by 19
  4. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraintscited by 17
  5. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  6. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  7. 2025Training Versatile Coding Agents in Synthetic Environmentscited by 13
  8. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 9
  9. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  10. 2026ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?cited by 6
  11. 2026SWE-Milestone: Evaluating AI Agents on Continuous Software Evolutioncited by 6
  12. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolutioncited by 6
21 more
  1. 2025Evolving Excellence: Automated Optimization of LLM-based Agentscited by 4
  2. 2026FormulaCode: Evaluating Agentic Optimization on Large Codebasescited by 4
  3. 2025GA4GC: Greener Agent for Greener Code via Multi-Objective Configuration Optimizationcited by 3
  4. 2026SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineeringcited by 3
  5. 2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 2
  6. 2026Evaluating LLMs on Real-World Software Performance Optimizationcited by 2
  7. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 2
  8. 2026Lean Refactor: Multi-Objective Controllable Proof Optimization via Agentic Strategy Searchcited by 1
  9. 2026Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?cited by 1
  10. 2026PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarkscited by 1
  11. 2026SemRep: Generative Code Representation Learning with Code Transformationscited by 0
  12. 2026EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedbackcited by 0
  13. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
  14. 2026BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledgecited by 0
  15. 2026Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproductioncited by 0
  16. 2026Rethinking Code Performance Benchmarks for LLMscited by 0
  17. 2026PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimizationcited by 0
  18. 2026When Models Edit Too Much: On the Fidelity of Minimal Code Editscited by 0
  19. 2026Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimizationcited by 0
  20. 2026Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmarkcited by 0
  21. 2026SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interactioncited by 0