# Automated Benchmark Generation for Repository-Level Coding Tasks

arXiv: https://arxiv.org/abs/2503.07701 · HTML: https://mlsanity.com/paper/2503.07701/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [Competition-Level Problems are Effective LLM Evaluators](https://mlsanity.com/paper/2312.02143/) (2023)
- [DyPyBench: A Benchmark of Executable Python Software](https://mlsanity.com/paper/2403.00539/) (2024)

## Built on by (most cited first, 29 of 29)

- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale](https://mlsanity.com/paper/2602.23866/) (2026)
- [SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories](https://mlsanity.com/paper/2512.17419/) (2025)
- [Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?](https://mlsanity.com/paper/2602.11988/) (2026)
- [SWE-Universe: Scale Real-World Verifiable Environments to Millions](https://mlsanity.com/paper/2602.02361/) (2026)
- [Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents](https://mlsanity.com/paper/2510.25694/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)
- [Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation](https://mlsanity.com/paper/2510.24358/) (2025)
- [CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability](https://mlsanity.com/paper/2602.03012/) (2026)
- [EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configuration](https://mlsanity.com/paper/2601.16489/) (2026)
- [SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks](https://mlsanity.com/paper/2506.10954/) (2025)
- [Coding Agents Don't Know When to Act](https://mlsanity.com/paper/2605.07769/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines](https://mlsanity.com/paper/2606.19830/) (2026)
- [AutoBaxBuilder: Bootstrapping Code Security Benchmarking](https://mlsanity.com/paper/2512.21132/) (2025)
- [MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering](https://mlsanity.com/paper/2601.22859/) (2026)
- [When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation](https://mlsanity.com/paper/2509.26600/) (2025)
- [DockSmith: Scaling Reliable Coding Environments via an Agentic Docker Builder](https://mlsanity.com/paper/2602.00592/) (2026)
- [ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation](https://mlsanity.com/paper/2603.26137/) (2026)
- [SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities](https://mlsanity.com/paper/2609.33763/) (2026)
- [CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation](https://mlsanity.com/paper/2507.19904/) (2025)
- [PIPer: On-Device Environment Setup via Online Reinforcement Learning](https://mlsanity.com/paper/2509.25455/) (2025)
- [Evaluating LLM Agents on Automated Software Analysis Tasks](https://mlsanity.com/paper/2604.11270/) (2026)
- [SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories](https://mlsanity.com/paper/2609.29465/) (2026)
- [AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering](https://mlsanity.com/paper/2609.32662/) (2026)
- [Zero2Repo: Can Coding Agents Build Repositories from Scratch?](https://mlsanity.com/paper/2609.38269/) (2026)
- [SWE-Duel: Towards a Self-Scaling Arena for Coding Agents](https://mlsanity.com/paper/2610.12569/) (2026)
- [SWE-Stream: Scaling Executable Training Data for SWE Agents via Progressive Construction and Elastic Streaming](https://mlsanity.com/paper/2610.20193/) (2026)

## Abstract (arXiv)

Code Agent development is an extremely active research area, where a reliable performance metric is critical for tracking progress and guiding new developments. This demand is underscored by the meteoric rise in popularity of SWE-Bench. This benchmark challenges code agents to generate patches addressing GitHub issues given the full repository as context. The correctness of generated patches is then evaluated by executing a human-written test suite extracted from the repository after the issue's resolution. However, constructing benchmarks like SWE-Bench requires substantial manual effort to set up historically accurate execution environments for testing. Crucially, this severely limits the number of considered repositories, e.g., just 12 for SWE-Bench. Considering so few repositories, selected for their popularity runs the risk of leading to a distributional mismatch, i.e., the measured performance may not be representative of real-world scenarios potentially misguiding development efforts. In this work, we address this challenge and introduce SetUpAgent, a fully automated system capable of historically accurate dependency setup, test execution, and result parsing. Using SetUpAgent, we generate two new datasets: (i) SWEE-Bench an extended version of SWE-Bench encompassing hundreds of repositories, and (ii) SWA-Bench a benchmark focusing on applications rather than libraries. Comparing these datasets to SWE-Bench with respect to their characteristics and code agent performance, we find significant distributional differences, including lower issue description quality and detail level, higher fix complexity, and most importantly up to 40% lower agent success rates.