mlsanityData to 7 Oct 2026

AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents

arXiv 2512.22387 ↗First posted 26 Dec 2025cs.SE cs.AI cs.MAcited by 5

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

The rise of Large Language Models (LLMs) as coding agents promises to accelerate software development, but their impact on generated code reproducibility remains largely unexplored. This paper presents an empirical study investigating whether LLM-generated code can be executed successfully in a clean environment with only OS packages and using only the dependencies that the model specifies. We evaluate three state-of-the-art LLM coding agents (Claude Code, OpenAI Codex, and Gemini) across 300 projects generated from 100 standardized prompts in Python, JavaScript, and Java. We introduce a three-layer dependency framework (distinguishing between claimed, working, and runtime dependencies) to quantify execution reproducibility. Our results show that only 68.3% of projects execute out-of-the-box, with substantial variation across languages (Python 89.2%, Java 44.0%). We also find a 13.5 times average expansion from declared to actual runtime dependencies, revealing significant hidden dependencies.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 8 of its 18 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

8 of 18 references.

References · 8

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  4. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  5. 2017An Empirical Comparison of Dependency Network Evolution in Seven Software Packaging Ecosystemscited by 20
  6. 2025Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Taskscited by 19
  7. 2024A Reproducibility and Generalizability Study of Large Language Models for Query Generationcited by 10
  8. 2017Sciunits: Reusable Research Objectscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026A Hierarchical Multi-Agent System for Autonomous Discovery in Geoscientific Data Archivescited by 4
  2. 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Qualitycited by 3
  3. 2026TDD Governance for Multi-Agent Code Generation via Prompt Engineeringcited by 1
  4. 2026RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescuecited by 1
  5. 2026From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
  2. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
  3. 2025A Survey on Code Generation with LLM-based Agents
  4. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
  5. 2026Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
  6. 2025Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation
  7. 2026RepoZero: Can LLMs Generate a Code Repository from Scratch?
  8. 2025Evaluating Software Process Models for Multi-Agent Class-Level Code Generation