mlsanityData to 7 Oct 2026

ProgramBench: Can Language Models Rebuild Programs From Scratch?

arXiv 2605.03546 ↗First posted 5 May 2026cs.SE cs.AIcited by 47

Not on the map. Why?

Cited by 36 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

27 of 41 references.

References · 27

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  6. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  7. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  8. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 127
  9. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  10. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  11. 2024Evaluating Language Models for Efficient Code Generationcited by 69
  12. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
15 more
  1. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 56
  2. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  3. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  4. 2025Repo2Run: Automated Building Executable Environment for Code Repository at Scalecited by 39
  5. 2024LLM4Decompile: Decompiling Binary Code with Large Language Modelscited by 38
  6. 2025EnvBench: A Benchmark for Automated Environment Setupcited by 36
  7. 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 36
  8. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  9. 2025SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?cited by 33
  10. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  11. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 25
  12. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  13. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  14. 2025CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesiscited by 15
  15. 2026Effective Strategies for Asynchronous Software Engineering Agentscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 47.

47 papers.

Sort:

By area:

  1. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  2. 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Taskscited by 9
  3. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 8
  4. 2026Decentralized Multi-Agent Systems with Shared Contextcited by 7
  5. 2026CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economiescited by 7
  6. 2026The Verification Horizon: No Silver Bullet for Coding Agent Rewardscited by 6
  7. 2026AgentRewind: Recoverable Execution for Long-Horizon LLM Agentscited by 6
  8. 2026ICAE-Bench: Evaluating Coding Agents as Interactive Project Builderscited by 5
  9. 2026MirrorCode: AI can rebuild entire programs from behavior alonecited by 4
  10. 2026SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Researchcited by 3
  11. 2026Position: Coding Benchmarks Are Misaligned with Agentic Software Engineeringcited by 3
  12. 2026Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolutioncited by 3
35 more
  1. 2026Cheap Code, Costly Judgment: A Case Study on Governable Agentic Software Engineeringcited by 2
  2. 2026ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenancecited by 2
  3. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 2
  4. 2026An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Modelscited by 1
  5. 2026Do programming languages still matter to your AI coding agent teammate? Evidence at scale from chess enginescited by 1
  6. 2026Can LLM Agents Infer World Models? Evidence from Agentic Automata Learningcited by 1
  7. 2026MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesiscited by 1
  8. 2026The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmarkcited by 1
  9. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  10. 2026RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agentscited by 1
  11. 2026CodeMidas: Scaling Agentic Coding RL Environments from Code Itselfcited by 1
  12. 2026Agensh: Scaling Organizational Intelligence to 1,024 Agentscited by 1
  13. 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systemscited by 0
  14. 2026ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?cited by 0
  15. 2026RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experimentscited by 0
  16. 2026SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratchcited by 0
  17. 2026SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvementscited by 0
  18. 2026Benchmarking General Mobile Assistants in Challenging Real-World Scenarioscited by 0
  19. 2026ττ-Bench: An Environment for End-To-End, Realistic Agent Constructioncited by 0
  20. 2026ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Taskscited by 0
  21. 2026Emergent Collusion in Long-Horizon LLM Agent Interactioncited by 0
  22. 2026OSFoundry: Building and Evolving Operating Systems with Specification-Guided Agentscited by 0
  23. 2026Evaluating Coding Agents on Kernel Exploit Generationcited by 0
  24. 2026SWE-Serve: Benchmarking Agentic Engineering For Production Inference Servingcited by 0
  25. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  26. 2026OptiArena: Can LLMs Improve Executable Algorithms under Fixed Resource Budgets?cited by 0
  27. 2026Can Agents Design Libraries for Agents?cited by 0
  28. 2026Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoningcited by 0
  29. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratchcited by 0
  30. 2026Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrailcited by 0
  31. 2026A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?cited by 0
  32. 2026AgentTime: Can Agents Estimate and Control Their Own Runtime?cited by 0
  33. 2026A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teachingcited by 0
  34. 2026MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvementcited by 0
  35. 2026Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model-Skill Co-Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
  2. 2025LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
  3. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
  4. 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills
  5. 2026ExplainBench: Evaluating Code Explanations from Agents
  6. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
  7. 2026ArchBench: Benchmarking Generative-AI for Software Architecture Tasks ↗
  8. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging