mlsanityData to 14 Oct 2026

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

arXiv 2509.16941 ↗First posted 21 Sep 2025cs.SE cs.CLcited by 279

Not on the map. Why?

Cited by 200 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level.

Key facts. Cited by 279 papers in the map (data through 14 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 21 Sep 2025.

Builds on

23 of 42 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  10. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  11. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  12. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
11 more
  1. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  3. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  4. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  5. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  6. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  7. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99
  8. 2025SWE-bench Goes Live!cited by 80
  9. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 42
  10. 2025SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?cited by 36
  11. 2025A Survey on Data Contamination for Large Language Modelscited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 270.

279 papers.

Sort:

By area:

  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  3. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 142
  4. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 95
  5. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 65
  6. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 59
  7. 2026DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Taskscited by 49
  8. 2026Code as Agent Harnesscited by 47
  9. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  10. 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 43
  11. 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 40
  12. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
38 more
  1. 2026Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interactioncited by 38
  2. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 34
  3. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 34
  4. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 28
  5. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 28
  6. 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
  7. 2026ContextBench: A Benchmark for Context Retrieval in Coding Agentscited by 25
  8. 2026SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agentscited by 25
  9. 2026SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integrationcited by 23
  10. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 23
  11. 2026SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Trainingcited by 22
  12. 2026Immersion in the GitHub Universe: Scaling Coding Agents to Masterycited by 22
  13. 2026BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?cited by 22
  14. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 19
  15. 2026Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learnedcited by 18
  16. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agentscited by 18
  17. 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architecturescited by 18
  18. 2026SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolutioncited by 18
  19. 2026Current Agents Fail to Leverage World Model as Tool for Foresightcited by 17
  20. 2026Effective Strategies for Asynchronous Software Engineering Agentscited by 16
  21. 2026AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Tracescited by 15
  22. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agentscited by 14
  23. 2025Training Versatile Coding Agents in Synthetic Environmentscited by 13
  24. 2026Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluationcited by 13
  25. 2026SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agentscited by 12
  26. 2026SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?cited by 12
  27. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 12
  28. 2025Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebasescited by 11
  29. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  30. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
  31. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 11
  32. 2026Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Modelcited by 11
  33. 2026BioAgent Bench: An AI Agent Evaluation Suite for Bioinformaticscited by 10
  34. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  35. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  36. 2026Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Developmentcited by 9
  37. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 9
  38. 2026Dockerless: Environment-Free Program Verifier for Coding Agentscited by 9

50 most cited of 279 All 279 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
  2. 2025Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
  3. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
  4. 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
  5. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints
  6. 2026SWE-Next: Scalable Real-World Software Engineering Tasks for Agents
  7. 2026ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
  8. 2025Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems