mlsanityData to 7 Oct 2026

Commit0: Library Generation from Scratch

arXiv 2412.01769 ↗First posted 2 Dec 2024cs.SE cs.AIcited by 50

Not on the map. Why?

Cited by 33 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the goal of benchmarking generative systems beyond expert software development ability, we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch. Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests, with the goal of producing an implementation of this API accordingly. The implementation is validated through running these unit tests. As a benchmark, Commit0 is designed to move beyond static one-shot code generation towards agents that must process long-form natural language specifications, adapt to multi-stage feedback, and generate code with complex dependencies. Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate. Our experiments demonstrate that while current agents can pass some unit tests, none can yet fully reproduce full libraries. Results also show that interactive feedback is quite useful for models to generate code that passes more unit tests, validating the benchmarks that facilitate its use.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

19 of 24 references.

References · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
7 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  3. 2021Measuring Coding Challenge Competence With APPScited by 879
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  5. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  6. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  7. 2024CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarkscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 30.

50 papers.

Sort:

By area:

  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  2. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  3. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 91
  4. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 56
  5. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 41
  6. 2025The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agentscited by 39
  7. 2025EnvBench: A Benchmark for Automated Environment Setupcited by 36
  8. 2026FeatureBench: Benchmarking Agentic Coding for Complex Feature Developmentcited by 30
  9. 2025SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuningcited by 27
  10. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 27
  11. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  12. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 24
38 more
  1. 2025HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimizationcited by 19
  2. 2025SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Mannercited by 18
  3. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 18
  4. 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Taskscited by 18
  5. 2026SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integrationcited by 18
  6. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  7. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 12
  8. 2026Effective Strategies for Asynchronous Software Engineering Agentscited by 12
  9. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  10. 2025Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modelingcited by 6
  11. 2026Hybrid-Gym: Training Coding Agents to Generalize Across Taskscited by 6
  12. 2026SWE-Milestone: Evaluating AI Agents on Continuous Software Evolutioncited by 6
  13. 2026Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarioscited by 5
  14. 2026ICAE-Bench: Evaluating Coding Agents as Interactive Project Builderscited by 5
  15. 2025Refactoring Codebases through Library Designcited by 4
  16. 2025How can we assess human-agent interactions? Case studies in software agent designcited by 4
  17. 2026FormulaCode: Evaluating Agentic Optimization on Large Codebasescited by 4
  18. 2026Multi-agent Collaboration with State Managementcited by 4
  19. 2026SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?cited by 4
  20. 2026Constraint Decay: The Fragility of LLM Agents in Backend Code Generationcited by 3
  21. 2026ATM: CID-Brokered Pre-Write Admission for Multi-Agent Code Co-Synthesiscited by 3
  22. 2026Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolutioncited by 2
  23. 2026Hodoscope: Unsupervised Monitoring for AI Misbehaviorscited by 2
  24. 2026Repo0: Design-Driven Zero-to-All Code Generationcited by 2
  25. 2026EarlyEval: Cheaper Agent Evaluation via Early Outcome Predictioncited by 2
  26. 2026Compiling Large Multi-Modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspectivecited by 1
  27. 2026EnterpriseLab: A Full-Stack Platform for developing and deploying agents in Enterprisescited by 1
  28. 2026Do programming languages still matter to your AI coding agent teammate? Evidence at scale from chess enginescited by 1
  29. 2026P3: Joint Program-and-Proof Planning for Verified Code Generationcited by 1
  30. 2026AuditRepairBench: A Paired-Execution Trace Corpus for Evaluator-Channel Ranking Instability in Agent Repaircited by 0
  31. 2026Stdlib or Third-Party? Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Librariescited by 0
  32. 2026Towards Evaluation of Implicit Software World Models in Coding LLMscited by 0
  33. 2026ττ-Bench: An Environment for End-To-End, Realistic Agent Constructioncited by 0
  34. 2026From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agentscited by 0
  35. 2026Can Agents Design Libraries for Agents?cited by 0
  36. 2026Zero2Repo: Can Coding Agents Build Repositories from Scratch?cited by 0
  37. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratchcited by 0
  38. 2026SecureVibe: Making Vibe Coding More Securecited by 0