# EnvBench: A Benchmark for Automated Environment Setup

arXiv: https://arxiv.org/abs/2503.14443 · HTML: https://mlsanity.com/paper/2503.14443/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 19 of 19)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models](https://mlsanity.com/paper/2402.02244/) (2024)
- [RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation](https://mlsanity.com/paper/2402.16667/) (2024)
- [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://mlsanity.com/paper/2311.09835/) (2023)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration](https://mlsanity.com/paper/2406.01422/) (2024)
- [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024)
- [Commit0: Library Generation from Scratch](https://mlsanity.com/paper/2412.01769/) (2024)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [RepoQA: Evaluating Long Context Code Understanding](https://mlsanity.com/paper/2406.06025/) (2024)
- [Sampling Projects in GitHub for MSR Studies](https://mlsanity.com/paper/2103.04682/) (2021)
- [Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects](https://mlsanity.com/paper/2412.06294/) (2024)
- [FlaPy: Mining Flaky Python Tests at Scale](https://mlsanity.com/paper/2305.04793/) (2023)
- [NLP Reproducibility For All: Understanding Experiences of Beginners](https://mlsanity.com/paper/2305.16579/) (2023)

## Built on by (most cited first, 30 of 36)

- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks](https://mlsanity.com/paper/2512.03262/) (2025)
- [SWE-Universe: Scale Real-World Verifiable Environments to Millions](https://mlsanity.com/paper/2602.02361/) (2026)
- [SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments](https://mlsanity.com/paper/2507.09063/) (2025)
- [Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents](https://mlsanity.com/paper/2510.25694/) (2025)
- [Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts](https://mlsanity.com/paper/2601.03315/) (2026)
- [Failure as a Process: An Anatomy of CLI Coding Agent Trajectories](https://mlsanity.com/paper/2607.09510/) (2026)
- [CODESKILL: Learning Self-Evolving Skills for Coding Agents](https://mlsanity.com/paper/2605.25430/) (2026)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [ClawEnvKit: Automatic Environment Generation for Claw-Like Agents](https://mlsanity.com/paper/2604.18543/) (2026)
- [SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks](https://mlsanity.com/paper/2506.10954/) (2025)
- [DeployBench: Benchmarking LLM Agents for Research Artifact Deployment](https://mlsanity.com/paper/2606.05238/) (2026)
- [SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle](https://mlsanity.com/paper/2605.13139/) (2026)
- [MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering](https://mlsanity.com/paper/2601.22859/) (2026)
- [DockSmith: Scaling Reliable Coding Environments via an Agentic Docker Builder](https://mlsanity.com/paper/2602.00592/) (2026)
- [ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution](https://mlsanity.com/paper/2603.06739/) (2026)
- [Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agents](https://mlsanity.com/paper/2602.09944/) (2026)
- [Self-Evolving Coding Agents](https://mlsanity.com/paper/2608.03392/) (2026)
- [RAT: RunAnyThing via Fully Automated Environment Configuration](https://mlsanity.com/paper/2604.23190/) (2026)
- [SetupX: Can LLM Agents Learn from Past Failures in Functionality-Correct Code Repository Setup?](https://mlsanity.com/paper/2605.26186/) (2026)
- [Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments](https://mlsanity.com/paper/2607.28591/) (2026)
- [QuoteBench: How Matched Scores Can Hide Command-Path Failures](https://mlsanity.com/paper/2608.13547/) (2026)
- [DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows](https://mlsanity.com/paper/2608.26546/) (2026)
- [PIPer: On-Device Environment Setup via Online Reinforcement Learning](https://mlsanity.com/paper/2509.25455/) (2025)
- [SW-A2-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web](https://mlsanity.com/paper/2604.04226/) (2026)
- [Evaluating LLM Agents on Automated Software Analysis Tasks](https://mlsanity.com/paper/2604.11270/) (2026)
- [BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledge](https://mlsanity.com/paper/2605.15815/) (2026)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have enabled researchers to focus on practical repository-level tasks in software engineering domain. In this work, we consider a cornerstone task for automating work with software repositories-environment setup, i.e., a task of configuring a repository-specific development environment on a system. Existing studies on environment setup introduce innovative agentic strategies, but their evaluation is often based on small datasets that may not capture the full range of configuration challenges encountered in practice. To address this gap, we introduce a comprehensive environment setup benchmark EnvBench. It encompasses 329 Python and 665 JVM-based (Java, Kotlin) repositories, with a focus on repositories that present genuine configuration challenges, excluding projects that can be fully configured by simple deterministic scripts. To enable further benchmark extension and usage for model tuning, we implement two automatic metrics: a static analysis check for missing imports in Python and a compilation check for JVM languages. We demonstrate the applicability of our benchmark by evaluating three environment setup approaches, including a simple zero-shot baseline and two agentic workflows, that we test with two powerful LLM backbones, GPT-4o and GPT-4o-mini. The best approach manages to successfully configure 6.69% repositories for Python and 29.47% repositories for JVM, suggesting that EnvBench remains challenging for current approaches. Our benchmark suite is publicly available at https://github.com/JetBrains-Research/EnvBench. The dataset and experiment trajectories are available at https://jb.gg/envbench.