mlsanityData to 7 Oct 2026

SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents

arXiv 2505.20411 ↗First posted 26 May 2025cs.SE cs.CLcited by 127

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM-based agents have shown promising capabilities in a growing range of software engineering (SWE) tasks. However, advancing this field faces two critical challenges. First, high-quality training data is scarce, especially data that reflects real-world SWE scenarios, where agents must interact with development environments, execute code and adapt behavior based on the outcomes of their actions. Existing datasets are either limited to one-shot code generation or comprise small, manually curated collections of interactive tasks, lacking both scale and diversity. Second, the lack of fresh interactive SWE tasks affects evaluation of rapidly improving models, as static benchmarks quickly become outdated due to contamination issues. To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale. Additionally, we use continuous supply of fresh tasks collected using SWE-rebench methodology to build a contamination-free benchmark for agentic software engineering. We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 79 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

18 of 79 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents11 Apr
  2. 2025Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute31 Mar
  3. 2025DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal18 Mar
  4. 2024Training Software Engineering Agents and Verifiers with SWE-Gym30 Dec
  5. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement26 Oct
  6. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future5 Aug

Other references · 12

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  9. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 95
  10. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  11. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  12. 2025RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 104.

127 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  2. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
  6. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 91
  7. 2026Qwen3-Coder-Next Technical Reportcited by 60
  8. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  9. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
  10. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  11. 2026DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Taskscited by 41
  12. 2025Agentic Software Engineering: Foundational Pillars and a Research Roadmapcited by 40
38 more
  1. 2025Context as a Tool: Context Management for Long-Horizon SWE-Agentscited by 38
  2. 2025Kimi-Dev: Agentless Training as Skill Prior for SWE-Agentscited by 35
  3. 2025Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learningcited by 34
  4. 2026On Data Engineering for Scaling LLM Terminal Capabilitiescited by 34
  5. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  6. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 32
  7. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolvingcited by 30
  8. 2025BugPilot: Complex Bug Generation for Efficient Learning of SWE Skillscited by 28
  9. 2026Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environmentscited by 27
  10. 2025SWE-RM: Execution-free Feedback For Software Engineering Agentscited by 24
  11. 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scalecited by 24
  12. 2026Tmax: A simple recipe for terminal agentscited by 24
  13. 2026Immersion in the GitHub Universe: Scaling Coding Agents to Masterycited by 20
  14. 2026BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?cited by 20
  15. 2026SWE-Explore: Benchmarking How Coding Agents Explore Repositoriescited by 20
  16. 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?cited by 19
  17. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  18. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraintscited by 17
  19. 2026SWE-Universe: Scale Real-World Verifiable Environments to Millionscited by 16
  20. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  21. 2026daVinci-Dev: Agent-native Mid-training for Software Engineeringcited by 15
  22. 2025Does SWE-Bench-Verified Test Agent Ability or Model Memory?cited by 14
  23. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  24. 2026daVinci-Env: Open SWE Environment Synthesis at Scalecited by 13
  25. 2026AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluationcited by 13
  26. 2026SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agentscited by 12
  27. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
  28. 2026Self-Compacting Language Model Agentscited by 11
  29. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Modelscited by 10
  30. 2026SERA: Soft-Verified Efficient Repository Agentscited by 10
  31. 2026MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Eracited by 9
  32. 2026SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understandingcited by 9
  33. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 8
  34. 2026CodeTaste: Can LLMs Generate Human-Level Code Refactorings?cited by 8
  35. 2026SWE-Next: Scalable Real-World Software Engineering Tasks for Agentscited by 8
  36. 2026The Interplay of Harness Design and Post-Training in LLM Agentscited by 8
  37. 2026GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environmentscited by 8
  38. 2026Orchard: An Open-Source Agentic Modeling Frameworkcited by 7

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training
  2. 2026SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
  3. 2026SWE-World: Building Software Engineering Agents in Docker-Free Environments
  4. 2025SWE-smith: Scaling Data for Software Engineering Agents
  5. 2026SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents
  6. 2025Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs
  7. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling
  8. 2025Training Versatile Coding Agents in Synthetic Environments