mlsanityData to 14 Oct 2026

SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents

arXiv 2504.08703 ↗First posted 11 Apr 2025cs.SEcited by 85

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Coding agents powered by large language models have shown impressive capabilities in software engineering tasks, but evaluating their performance across diverse programming languages and real-world scenarios remains challenging. We introduce SWE-PolyBench, a new multi-language benchmark for repository-level, execution-based evaluation of coding agents. SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code refactoring. We provide a task and repository-stratified subsample (SWE-PolyBench500) and release an evaluation harness allowing for fully automated evaluation. To enable a more comprehensive comparison of coding agents, this work also presents a novel set of metrics rooted in syntax tree analysis. We evaluate leading open source coding agents on SWE-PolyBench, revealing their strengths and limitations across languages, task types, and complexity classes. Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks. SWE-PolyBench aims to drive progress in developing more versatile and robust AI coding assistants for real-world software engineering. Our datasets and code are available at: https://github.com/amazon-science/SWE-PolyBench

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 38 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

21 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 20

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2021Measuring Coding Challenge Competence With APPScited by 902
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  6. 2024A Survey on Large Language Models for Code Generationcited by 658
  7. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  8. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  9. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 112
  10. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  11. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  12. 2024DebugBench: Evaluating Debugging Capability of Large Language Modelscited by 76
8 more
  1. 2024Automated Unit Test Improvement using Large Language Models at Metacited by 75
  2. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 70
  3. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 65
  4. 2022Code Translation with Compiler Representationscited by 42
  5. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 42
  6. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  7. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 41
  8. 2024SWE-bench-java: A GitHub Issue Resolving Benchmark for Javacited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 61.

85 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  2. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
  3. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 95
  4. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 75
  5. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 59
  6. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 30
  7. 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scalecited by 29
  8. 2025When Agents go Astray: Course-Correcting SWE Agents with PRMscited by 28
  9. 2026ContextBench: A Benchmark for Context Retrieval in Coding Agentscited by 25
  10. 2026Immersion in the GitHub Universe: Scaling Coding Agents to Masterycited by 22
  11. 2026BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?cited by 22
  12. 2026SWE-Universe: Scale Real-World Verifiable Environments to Millionscited by 19
38 more
  1. 2025Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluationcited by 15
  2. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  3. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  4. 2025CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoningcited by 13
  5. 2025Can Language Models Discover Scaling Laws?cited by 13
  6. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  7. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Modelscited by 11
  8. 2026Open-SWE-Traces: Advancing Dual-Mode Multilingual Distillation for Software Engineering Agentscited by 10
  9. 2026Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agentscited by 8
  10. 2026SkillMaster: Toward Autonomous Skill Mastery in LLM Agentscited by 8
  11. 2026Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agentscited by 8
  12. 2025FeatBench: Towards More Realistic Evaluation of Feature-level Code Generationcited by 6
  13. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  14. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 6
  15. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  16. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoringcited by 6
  17. 2025SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solvingcited by 5
  18. 2025CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistancecited by 5
  19. 2025CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournamentscited by 5
  20. 2025SweRank+: Multilingual, Multi-Turn Code Ranking for Software Issue Localizationcited by 5
  21. 2026REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usagecited by 5
  22. 2026SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Testscited by 5
  23. 2026What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agentscited by 5
  24. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  25. 2026OmniCode: A Benchmark for Evaluating Software Engineering Agentscited by 4
  26. 2026ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluationcited by 3
  27. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Reviewcited by 3
  28. 2026Constraint Decay: The Fragility of LLM Agents in Backend Code Generationcited by 3
  29. 2026RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbationscited by 3
  30. 2026Loc2Repair: A Framework for Evaluating the Impact of File-Level Issue Localization in Repo-Level LLM Repaircited by 3
  31. 2026Locating Hidden Failures Makes Long-Horizon Agents More Reliablecited by 3
  32. 2025TENET: One Step Toward Test-Driven Development for Repository-Level Code Generationcited by 2
  33. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  34. 2026HARP: Measuring Harm Amplification in Multi-Agent LLM Systemscited by 2
  35. 2026Code Isn't Memory: A Structural Codebase Index Inside a Coding Agentcited by 2
  36. 2026A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agentscited by 2
  37. 2026ττ-Bench: An Environment for End-To-End, Realistic Agent Constructioncited by 2
  38. 2025Integrating Performance Tools in Model Reasoning for GPU Kernel Optimizationcited by 1

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SWE-Explore: Benchmarking How Coding Agents Explore Repositories
  2. 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
  3. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
  4. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
  5. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
  6. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
  7. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
  8. 2026FeatureBench: Benchmarking Agentic Coding for Complex Feature Development