mlsanityData to 14 Oct 2026

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

arXiv 2310.06770 ↗First posted 10 Oct 2023cs.CL cs.AI cs.SEcited by 3,373

Not on the map. Why?

Cited by 1,829 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.

Key facts. Cited by 3,373 papers in the map (data through 14 Oct 2026). Builds on 51 of its 64 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

51 of 64 references.

References · 51

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
38 more
  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2021Measuring Coding Challenge Competence With APPScited by 902
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  4. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  5. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  6. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  7. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  8. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  9. 2017Learning to Represent Programs with Graphscited by 220
  10. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  11. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  12. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  13. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 166
  14. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  15. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  16. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  17. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  18. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  19. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  20. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  21. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  22. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  23. 2022Automated Repair of Programs from Large Language Modelscited by 71
  24. 2023Towards an Understanding of Large Language Models in Software Engineering Taskscited by 68
  25. 2022CoditT5: Pretraining for Source Code and Natural Language Editingcited by 45
  26. 2023Conversational Automated Program Repaircited by 41
  27. 2019How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Datasetcited by 36
  28. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  29. 2023Large Language Models of Code Fail at Completing Code with Potential Bugscited by 29
  30. 2023CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Modelscited by 28
  31. 2021On Multi-Modal Learning of Editing Source Codecited by 25
  32. 2018Automatic Software Repair: a Bibliographycited by 24
  33. 2021CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Modelcited by 16
  34. 2023Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptionscited by 14
  35. 2020Better Automatic Program Repair by Using Bug Reports and Tests Togethercited by 10
  36. 2019Language Tasks and Language Games: On Methodology in Current Natural Language Processing Researchcited by 6
  37. 2022CommitBART: A Large Pre-trained Model for GitHub Commitscited by 6
  38. 2018Entropy Guided Spectrum Based Bug Localization Using Statistical Language Modelcited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 236; 2025: 802; 2026: 2325.

3,373 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  3. 2024OpenAI o1 System Cardcited by 1,244
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  5. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  7. 2024A Survey on Large Language Models for Code Generationcited by 658
  8. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  9. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  10. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  11. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  12. 2025Kimi K2: Open Agentic Intelligencecited by 567
38 more
  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  4. 2025Humanity's Last Examcited by 491
  5. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  7. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  8. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  9. 2024Automated Design of Agentic Systemscited by 407
  10. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  12. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  13. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  14. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  15. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  16. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 315
  17. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  18. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  19. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  20. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  21. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  22. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  23. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  24. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  25. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  26. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  27. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  28. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  29. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 226
  30. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  31. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
  32. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  33. 2025Why Language Models Hallucinatecited by 220
  34. 2024AI Agents That Mattercited by 219
  35. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  36. 2025Survey on Evaluation of LLM-based Agentscited by 204
  37. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  38. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191

50 most cited of 3,373 All 3,373 (JSON)