mlsanityData to 14 Oct 2026

Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving

arXiv 2504.02605 ↗First posted 3 Apr 2025cs.SE cs.AI cs.CLcited by 171

Not on the map. Why?

Cited by 120 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue. However, existing benchmarks, such as SWE-bench, focus almost exclusively on Python, making them insufficient for evaluating Large Language Models (LLMs) across diverse software ecosystems. To address this, we introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a total of 1,632 high-quality instances, which were carefully annotated from 2,456 candidates by 68 expert annotators, ensuring that the benchmark can provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we evaluate a series of state-of-the-art models using three representative methods (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with key empirical insights. In addition, we launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning (RL) training datasets for issue-resolving tasks. As an initial contribution, we release a set of 4,723 well-structured instances spanning seven programming languages, laying a solid foundation for RL research in this domain. More importantly, we open-source our entire data production pipeline, along with detailed tutorials, encouraging the open-source community to continuously contribute and expand the dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL community as catalysts for advancing RL toward its full potential, bringing us one step closer to the dawn of AGI.

Key facts. Cited by 171 papers in the map (data through 14 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

30 of 49 references.

References · 30

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  5. 2024A Survey on Large Language Models for Code Generationcited by 658
  6. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  7. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  9. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  10. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  11. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  12. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
18 more
  1. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  2. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  3. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  4. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 133
  5. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  6. 2018Mapping Language to Code in Programmatic Contextcited by 104
  7. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  8. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  9. 2023A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trendscited by 66
  10. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 46
  11. 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
  12. 2024A Deep Dive into Large Language Models for Automated Bug Localization and Repaircited by 39
  13. 2025EnvBench: A Benchmark for Automated Environment Setupcited by 36
  14. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  15. 2024AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptionscited by 16
  16. 2025An Empirical Study on Leveraging Images in Automated Bug Report Reproductioncited by 4
  17. 2023GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actionscited by 3
  18. 2024Exploring the Integration of Large Language Models in Industrial Test Maintenance Processescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 42; 2026: 129.

171 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 567
  2. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  3. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  4. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  5. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 95
  6. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 75
  7. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 65
  8. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 61
  9. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Predictioncited by 59
  10. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 59
  11. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  12. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 50
38 more
  1. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  2. 2025Reducing Cost of LLM Agents with Trajectory Reductioncited by 48
  3. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 48
  4. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
  5. 2025Context as a Tool: Context Management for Long-Horizon SWE-Agentscited by 43
  6. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 41
  7. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 34
  8. 2026Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environmentscited by 33
  9. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 30
  10. 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scalecited by 29
  11. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 25
  12. 2026ContextBench: A Benchmark for Context Retrieval in Coding Agentscited by 25
  13. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 23
  14. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 23
  15. 2026Immersion in the GitHub Universe: Scaling Coding Agents to Masterycited by 22
  16. 2026BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?cited by 22
  17. 2026Recursive Synthesis for Long-Horizon Terminal Taskscited by 19
  18. 2026Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learnedcited by 18
  19. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraintscited by 17
  20. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learningcited by 17
  21. 2025Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Studycited by 16
  22. 2025SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applicationscited by 16
  23. 2025An Empirical Study on Failures in Automated Issue Solvingcited by 16
  24. 2025Does SWE-Bench-Verified Test Agent Ability or Model Memory?cited by 16
  25. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  26. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  27. 2026A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compressioncited by 15
  28. 2026AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluationcited by 15
  29. 2025TOM-SWE: User Mental Modeling For Software Engineering Agentscited by 13
  30. 2025Training Versatile Coding Agents in Synthetic Environmentscited by 13
  31. 2025SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimationcited by 12
  32. 2026SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?cited by 12
  33. 2026Structurally Aligned Subtask-Level Memory for Software Engineering Agentscited by 12
  34. 2026DevOps-Gym: Benchmarking AI Agents in Software DevOps Cyclecited by 11
  35. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failurecited by 11
  36. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 11
  37. 2026SERA: Soft-Verified Efficient Repository Agentscited by 10
  38. 2025SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Taskscited by 9

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SWE-Exp: Experience-Driven Software Issue Resolution
  2. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving
  3. 2025SweRank+: Multilingual, Multi-Turn Code Ranking for Software Issue Localization
  4. 2025The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
  5. 2026Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey
  6. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
  7. 2026ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance
  8. 2025SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution