# PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization

arXiv: https://arxiv.org/abs/2605.15222 · HTML: https://mlsanity.com/paper/2605.15222/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 21 of 21)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [A Survey on Code Generation with LLM-based Agents](https://mlsanity.com/paper/2508.00083/) (2025)
- [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [EffiBench: Benchmarking the Efficiency of Automatically Generated Code](https://mlsanity.com/paper/2402.02037/) (2024)
- [FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation](https://mlsanity.com/paper/2503.06680/) (2025)
- [Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks](https://mlsanity.com/paper/2505.07473/) (2025)
- [EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code](https://mlsanity.com/paper/2505.13004/) (2025)
- [ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation](https://mlsanity.com/paper/2503.07010/) (2025)
- [AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators](https://mlsanity.com/paper/2508.09101/) (2025)
- [COFFE: A Code Efficiency Benchmark for Code Generation](https://mlsanity.com/paper/2502.02827/) (2025)
- [CodeArena: A Collective Evaluation Platform for LLM Code Generation](https://mlsanity.com/paper/2503.01295/) (2025)
- [LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities](https://mlsanity.com/paper/2601.09822/) (2026)
- [Parsing Gigabytes of JSON per Second](https://mlsanity.com/paper/1902.08318/) (2019)
- [RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices](https://mlsanity.com/paper/2604.22659/) (2026)

## Built on by (most cited first, 2 of 2)

- [Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?](https://mlsanity.com/paper/2607.01211/) (2026)
- [RLPF: Reinforcement Learning from Performance Feedback for Code Generation](https://mlsanity.com/paper/2607.27271/) (2026)

## Abstract (arXiv)

Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks. Each task includes executable correctness checks, a baseline implementation, and a reference optimized solution. This allows us to evaluate both correctness and runtime-oriented efficiency. Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations. Current models also show weaknesses in cross-language robustness and in consistently reaching expert-level efficiency. These results suggest that performance-aware evaluation are still needed. LLMs should move beyond generating merely correct code toward producing efficient systems software. We submit the benchmark data, evaluation infrastructure, and complete logs of all LLMs-generated code at https://anonymous.4open.science/r/perfcodebench-7CDE.