# RepoZero: Can LLMs Generate a Code Repository from Scratch?

arXiv: https://arxiv.org/abs/2605.07122 · HTML: https://mlsanity.com/paper/2605.07122/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Built on by (most cited first, 14 of 14)

- [The Verification Horizon: No Silver Bullet for Coding Agent Rewards](https://mlsanity.com/paper/2606.26300/) (2026)
- [SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research](https://mlsanity.com/paper/2606.09730/) (2026)
- [MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis](https://mlsanity.com/paper/2607.27146/) (2026)
- [ττ-Bench: An Environment for End-To-End, Realistic Agent Construction](https://mlsanity.com/paper/2609.04611/) (2026)
- [CodeMidas: Scaling Agentic Coding RL Environments from Code Itself](https://mlsanity.com/paper/2609.22068/) (2026)
- [ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?](https://mlsanity.com/paper/2606.19787/) (2026)
- [Stdlib or Third-Party? Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Libraries](https://mlsanity.com/paper/2605.21405/) (2026)
- [Forward-Deployed Full-Stack Engineering for Autonomous Cloud MLOps](https://mlsanity.com/paper/2608.29615/) (2026)
- [Can Agents Design Libraries for Agents?](https://mlsanity.com/paper/2609.36730/) (2026)
- [Zero2Repo: Can Coding Agents Build Repositories from Scratch?](https://mlsanity.com/paper/2609.38269/) (2026)
- [E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch](https://mlsanity.com/paper/2609.38335/) (2026)
- [Large-scale Repository Engineering via Agent-Native Reusable Code Primitives](https://mlsanity.com/paper/2610.09079/) (2026)
- [SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction](https://mlsanity.com/paper/2610.11559/) (2026)
- [Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents](https://mlsanity.com/paper/2610.20344/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have recently shown remarkable progress in code generation, yet their ability to construct complete software repositories from scratch remains poorly understood. A fundamental bottleneck is the lack of verifiable and scalable evaluation: existing benchmarks either focus on patch-based editing or rely on human or LLM-based judgments, which introduce bias and limit reproducibility. In this work, we present RepoZero, the first benchmark that enables fully automated, execution-based verification of repository-level generation from scratch. Our key idea is to reformulate generation as repository reproduction: given only API specifications, an agent must re-implement an entire repository such that its behavior matches the original implementation. This design allows for strict black-box validation via output equivalence, while naturally supporting large-scale construction by reusing existing open-source repositories. To further mitigate data leakage and shortcut solutions, we introduce cross-language constraints and a sandboxed evaluation protocol. Building on this benchmark, we propose an Agentic Code-Test Evolution (ACE) framework that performs iterative test generation and error-driven refinement, enabling effective test-time scaling for repository-level synthesis. Extensive experiments across multiple state-of-the-art LLMs and agent frameworks reveal that even the strongest LLM agents achieve only limited pass rates (30% - 55%), exposing a substantial gap between current capabilities and real-world software development requirements. Our results establish RepoZero as a challenging, scalable, and reliable testbed for end-to-end code generation, and highlight self-verification via test generation as a critical direction for advancing LLM-based coding agents.