mlsanityData to 7 Oct 2026

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

arXiv 2406.12952 ↗First posted 18 Jun 2024cs.SE cs.AI cs.LGcited by 129

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests. We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at https://github.com/logic-star-ai/SWT-Bench

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 49 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

21 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution26 Mar
  2. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repair25 Mar

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2022CodeT: Code Generation with Generated Testscited by 369
  10. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  11. 2024AutoCodeRover: Autonomous Program Improvementcited by 272
  12. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
7 more
  1. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  2. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  3. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 94
  4. 2024Automated Unit Test Improvement using Large Language Models at Metacited by 73
  5. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  6. 2023Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Promptingcited by 23
  7. 2023A Survey on What Developers Think About Testingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 53; 2026: 72.

129 papers.

Sort:

By area:

  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  3. 2025Survey on Evaluation of LLM-based Agentscited by 190
  4. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  5. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  6. 2025LocAgent: Graph-Guided LLM Agents for Code Localizationcited by 82
  7. 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectoriescited by 75
  8. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  9. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  10. 2025SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolutioncited by 52
  11. 2025The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agentscited by 39
  12. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
38 more
  1. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 32
  2. 2025Type-Constrained Code Generation with Language Modelscited by 32
  3. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  4. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 29
  5. 2025Automated Benchmark Generation for Repository-Level Coding Taskscited by 27
  6. 2025Otter: Generating Tests from Issues to Validate SWE Patchescited by 24
  7. 2026TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agentscited by 24
  8. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Codingcited by 22
  9. 2025Agentic Bug Reproduction for Effective Automated Program Repair at Googlecited by 20
  10. 2025Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Taskscited by 20
  11. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 20
  12. 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?cited by 19
  13. 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Taskscited by 18
  14. 2025Unified Software Engineering Agent as AI Software Engineercited by 17
  15. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraintscited by 17
  16. 2025Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Studycited by 16
  17. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  18. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  19. 2025Towards LLM Agents for Earth Observationcited by 13
  20. 2025Constrained Decoding of Diffusion LLMs with Context-Free Grammarscited by 13
  21. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  22. 2025Training Versatile Coding Agents in Synthetic Environmentscited by 13
  23. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  24. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 11
  25. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
  26. 2024LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issuescited by 10
  27. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  28. 2025Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Taskscited by 9
  29. 2025Self-Abstraction from Grounded Experience for Plan-Guided Policy Refinementcited by 9
  30. 2026DevOps-Gym: Benchmarking AI Agents in Software DevOps Cyclecited by 9
  31. 2026WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Gamescited by 9
  32. 2025Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoningcited by 8
  33. 2025Benchmarking LLMs for Unit Test Generation from Real-World Functionscited by 8
  34. 2026CodeTaste: Can LLMs Generate Human-Level Code Refactorings?cited by 8
  35. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  36. 2025CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verificationcited by 7
  37. 2025InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolutioncited by 7
  38. 2026Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agentscited by 7

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TDD-Agent: Test-Driven Reasoning for Code Generation
  2. 2024Large Language Models as Test Case Generators: Performance Evaluation and Enhancement
  3. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
  4. 2026SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
  5. 2024An Empirical Study on LLM-based Agents for Automated Bug Fixing
  6. 2025UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
  7. 2026Agent-Based Test Assertion Generation via Diverse Perspective Aggregation
  8. 2024MarsCode Agent: AI-native Automated Bug Fixing