# Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects

arXiv: https://arxiv.org/abs/2412.06294 · HTML: https://mlsanity.com/paper/2412.06294/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023)
- [MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution](https://mlsanity.com/paper/2403.17927/) (2024)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022)
- [Thread of Thought Unraveling Chaotic Contexts](https://mlsanity.com/paper/2311.08734/) (2023)
- [A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization](https://mlsanity.com/paper/2308.05487/) (2023)
- [Explainable Automated Debugging via Large Language Model-driven Scientific Debugging](https://mlsanity.com/paper/2304.02195/) (2023)
- [Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation](https://mlsanity.com/paper/2401.06391/) (2024)
- [Neural Models for Information Retrieval](https://mlsanity.com/paper/1705.01509/) (2017)
- [Can GPT-4 Replicate Empirical Software Engineering Research?](https://mlsanity.com/paper/2310.01727/) (2023)
- [Automatic Detection of Five API Documentation Smells: Practitioners' Perspectives](https://mlsanity.com/paper/2102.08486/) (2021)

## Built on by (most cited first, 21 of 21)

- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [EnvBench: A Benchmark for Automated Environment Setup](https://mlsanity.com/paper/2503.14443/) (2025)
- [SWE-Universe: Scale Real-World Verifiable Environments to Millions](https://mlsanity.com/paper/2602.02361/) (2026)
- [SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments](https://mlsanity.com/paper/2507.09063/) (2025)
- [Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents](https://mlsanity.com/paper/2510.25694/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [AgentStepper: Interactive Debugging of Software Development Agents](https://mlsanity.com/paper/2602.06593/) (2026)
- [EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configuration](https://mlsanity.com/paper/2601.16489/) (2026)
- [DeployBench: Benchmarking LLM Agents for Research Artifact Deployment](https://mlsanity.com/paper/2606.05238/) (2026)
- [MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering](https://mlsanity.com/paper/2601.22859/) (2026)
- [ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution](https://mlsanity.com/paper/2603.06739/) (2026)
- [BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software](https://mlsanity.com/paper/2509.25248/) (2025)
- [Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair](https://mlsanity.com/paper/2511.11012/) (2025)
- [Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agents](https://mlsanity.com/paper/2602.09944/) (2026)
- [RAT: RunAnyThing via Fully Automated Environment Configuration](https://mlsanity.com/paper/2604.23190/) (2026)
- [PIPer: On-Device Environment Setup via Online Reinforcement Learning](https://mlsanity.com/paper/2509.25455/) (2025)
- [Evaluating LLM Agents on Automated Software Analysis Tasks](https://mlsanity.com/paper/2604.11270/) (2026)
- [BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledge](https://mlsanity.com/paper/2605.15815/) (2026)
- [FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration](https://mlsanity.com/paper/2609.27571/) (2026)
- [RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?](https://mlsanity.com/paper/2609.28850/) (2026)
- [Graph-Guided Repository Environment Construction](https://mlsanity.com/paper/2609.33429/) (2026)

## Abstract (arXiv)

Many works have recently proposed the use of Large Language Model (LLM) based agents for performing `repository level' tasks, loosely defined as a set of tasks whose scopes are greater than a single file. This has led to speculation that the orchestration of these repository-level tasks could lead to software engineering agents capable of performing almost independently of human intervention. However, of the suite of tasks that would need to be performed by this autonomous software engineering agent, we argue that one important task is missing, which is to fulfil project level dependency by installing other repositories. To investigate the feasibility of this repository level installation task, we introduce a benchmark of of repository installation tasks curated from 40 open source Python projects, which includes a ground truth installation process for each target repository. Further, we propose Installamatic, an agent which aims to perform and verify the installation of a given repository by searching for relevant instructions from documentation in the repository. Empirical experiments reveal that that 55% of the studied repositories can be automatically installed by our agent at least one out of ten times. Through further analysis, we identify the common causes for our agent's inability to install a repository, discuss the challenges faced in the design and implementation of such an agent and consider the implications that such an agent could have for developers.