mlsanityData to 14 Oct 2026

GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents

arXiv 2505.23671 ↗First posted 29 May 2025cs.SE cs.AI cs.CLcited by 25

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Developing high-performance software is a complex task that requires specialized expertise. We introduce GSO, a benchmark for evaluating language models' capabilities in developing high-performance software. We develop an automated pipeline that generates and executes performance tests to analyze repository commit histories to identify 102 challenging optimization tasks across 10 codebases, spanning diverse domains and programming languages. An agent is provided with a codebase and performance test as a precise specification, and tasked to improve the runtime efficiency, which is measured against the expert developer optimization. Our quantitative evaluation reveals that leading SWE-Agents struggle significantly, achieving less than 5% success rate, with limited improvements even with inference-time scaling. Our qualitative analysis identifies key failure modes, including difficulties with low-level languages, practicing lazy optimization strategies, and challenges in accurately localizing bottlenecks. We release the code and artifacts of our benchmark along with agent trajectories to enable future research.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 57 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

43 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents9 Apr
  2. 2025LocAgent: Graph-Guided LLM Agents for Code Localization12 Mar
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul

Other references · 40

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2021Measuring Coding Challenge Competence With APPScited by 902
  11. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  12. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
28 more
  1. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  2. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  3. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  4. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  5. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  6. 2023Learning Performance-Improving Code Editscited by 106
  7. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  8. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  9. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99
  10. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  11. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  12. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 83
  13. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  14. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 64
  15. 2024TestGenEval: A Real World Unit Test Generation and Test Completion Benchmarkcited by 52
  16. 2024Commit0: Library Generation from Scratchcited by 52
  17. 2024A Performance Study of LLM-Generated Code on Leetcodecited by 49
  18. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  19. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 34
  20. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 27
  21. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 26
  22. 2024CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarkscited by 23
  23. 2024On Evaluating the Efficiency of Source Code Generated by LLMscited by 21
  24. 2021Meta Back-translationcited by 16
  25. 2023Explaining Competitive-Level Programming Solutions using LLMscited by 13
  26. 2025RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testingcited by 13
  27. 2025BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?cited by 12
  28. 2025Challenges and Paths Towards AI for Software Engineeringcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 18.

25 papers.

Sort:

By area:

  1. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 59
  2. 2025A Rosetta Stone for AI Benchmarkscited by 36
  3. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  4. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 27
  5. 2026Immersion in the GitHub Universe: Scaling Coding Agents to Masterycited by 22
  6. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 21
  7. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  8. 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 11
  9. 2026Open-SWE-Traces: Advancing Dual-Mode Multilingual Distillation for Software Engineering Agentscited by 10
  10. 2026SWE-Milestone: Evaluating AI Agents on Continuous Software Evolutioncited by 8
  11. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  12. 2026ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?cited by 6
13 more
  1. 2026Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Taskscited by 6
  2. 2025Optimizing PyTorch Inference with LLM-Based Multi-Agent Systemscited by 4
  3. 2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 4
  4. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 3
  5. 2026Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?cited by 1
  6. 2026Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarkscited by 1
  7. 2026SemRep: Generative Code Representation Learning with Code Transformationscited by 0
  8. 2026EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedbackcited by 0
  9. 2026Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluationcited by 0
  10. 2026Predicting Task Difficulty Without Rolloutscited by 0
  11. 2026What Makes Software Issue Resolution Tasks Difficult for Agents?cited by 0
  12. 2026One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agentscited by 0
  13. 2026Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
  2. 2026SWE-World: Building Software Engineering Agents in Docker-Free Environments
  3. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
  4. 2026PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization
  5. 2026SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents
  6. 2026FormulaCode: Evaluating Agentic Optimization on Large Codebases
  7. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
  8. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents