mlsanityData to 14 Oct 2026

SWE-bench-java: A GitHub Issue Resolving Benchmark for Java

arXiv 2408.14354 ↗First posted 26 Aug 2024cs.SE cs.AI cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

GitHub issue resolving is a critical task in software engineering, recently gaining significant attention in both industry and academia. Within this task, SWE-bench has been released to evaluate issue resolving capabilities of large language models (LLMs), but has so far only focused on Python version. However, supporting more programming languages is also important, as there is a strong demand in industry. As a first step toward multilingual support, we have developed a Java version of SWE-bench, called SWE-bench-java. We have publicly released the dataset, along with the corresponding Docker-based evaluation environment and leaderboard, which will be continuously maintained and updated in the coming months. To verify the reliability of SWE-bench-java, we implement a classic method SWE-agent and test several powerful LLMs on it. As is well known, developing a high-quality multi-lingual benchmark is time-consuming and labor-intensive, so we welcome contributions through pull requests or collaboration to accelerate its iteration and refinement, paving the way for fully automated programming.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

20 of 32 references.

References · 20

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  5. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  6. 2022CodeT: Code Generation with Generated Testscited by 384
  7. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  9. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  10. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  11. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  12. 2022Large Language Models Meet NL2Code: A Surveycited by 138
8 more
  1. 2018Mapping Language to Code in Programmatic Contextcited by 104
  2. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  3. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  4. 2023A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trendscited by 66
  5. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 65
  6. 2022When Language Model Meets Private Librarycited by 54
  7. 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
  8. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 15.

34 papers.

Sort:

By area:

  1. 2025Survey on Evaluation of LLM-based Agentscited by 204
  2. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 85
  3. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 34
  4. 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scalecited by 29
  5. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  6. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  7. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 18
  8. 2025Does SWE-Bench-Verified Test Agent Ability or Model Memory?cited by 16
  9. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  10. 2025An Empirical Study of Agent Developer Practices in AI Agent Frameworkscited by 14
  11. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  12. 2024CodeV: Issue Resolving with Visual Datacited by 9
22 more
  1. 2025Can Agents Fix Agent Issues?cited by 9
  2. 2025SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarkscited by 7
  3. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  4. 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Developmentcited by 6
  5. 2025SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solvingcited by 5
  6. 2025EffiReasonTrans: RL-Optimized Reasoning for Code Translationcited by 5
  7. 2026RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Modelscited by 5
  8. 2025Programming with Pixels: Can Computer-Use Agents do Software Engineering?cited by 4
  9. 2026OmniCode: A Benchmark for Evaluating Software Engineering Agentscited by 4
  10. 2025SAEL: Leveraging Large Language Models with Adaptive Mixture-of-Experts for Smart Contract Vulnerability Detectioncited by 2
  11. 2026Yet Even Less Is Even Better For Agentic, Reasoning, and Coding LLMscited by 2
  12. 2025BackportBench: A Multilingual Benchmark for Automated Backporting of Patchescited by 1
  13. 2026How Much Static Structure Do Code Agents Need? A Study of Deterministic Anchoringcited by 1
  14. 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
  15. 2026SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agentscited by 1
  16. 2026Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineeringcited by 0
  17. 2026GREPO: A Benchmark for Graph Neural Networks on Repository-Level Bug Localizationcited by 0
  18. 2026MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Developmentcited by 0
  19. 2026Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMscited by 0
  20. 2026SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositoriescited by 0
  21. 2026SWE-PolyVision: Benchmarking Cross-Image Abductive Reasoning for Repository-Level Software Engineeringcited by 0
  22. 2026Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agentscited by 0