mlsanityData to 14 Oct 2026

You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects

arXiv 2412.10133 ↗First posted 13 Dec 2024cs.SE cs.AIcited by 47

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

The ability to execute the test suite of a project is essential in many scenarios, e.g., to assess code quality and code coverage, to validate code changes made by developers or automated tools, and to ensure compatibility with dependencies. Despite its importance, executing the test suite of a project can be challenging in practice because different projects use different programming languages, software ecosystems, build systems, testing frameworks, and other tools. These challenges make it difficult to create a reliable, universal test execution method that works across different projects. This paper presents ExecutionAgent, an automated technique that prepares scripts for building an arbitrary project from source code and running its test cases. Inspired by the way a human developer would address this task, our approach is a large language model (LLM)-based agent that autonomously executes commands and interacts with the host system. The agent uses meta-prompting to gather guidelines on the latest technologies related to the given project, and it iteratively refines its process based on feedback from the previous steps. Our evaluation applies ExecutionAgent to 50 open-source projects that use 14 different programming languages and many different build and testing tools. The approach successfully executes the test suites of 33/50 projects, while matching the test results of ground truth test suite executions with a deviation of only 7.5%. These results improve over the best previously available technique by 6.6x. The costs imposed by the approach are reasonable, with an execution time of 74 minutes and LLM costs of USD 0.16, on average per project. We envision ExecutionAgent to serve as a valuable tool for developers, automated programming tools, and researchers that need to execute tests across a wide variety of projects.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 32 of its 49 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

32 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MarsCode Agent: AI-native Automated Bug Fixing2 Sep
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
  3. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution26 Mar
  4. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repair25 Mar
  5. 2024Exploring LLM-based Agents for Root Cause Analysis7 Mar

Other references · 27

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  6. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  7. 2020Unsupervised Translation of Programming Languagescited by 263
  8. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  9. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  10. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  11. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 111
  12. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
15 more
  1. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  2. 2023An In-Context Learning Agent for Formal Theorem-Provingcited by 82
  3. 2024Automated Unit Test Improvement using Large Language Models at Metacited by 75
  4. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 65
  5. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 64
  6. 2024Calibration and Correctness of Language Models for Codecited by 49
  7. 2023Prompting Is All You Need: Automated Android Bug Replay with Large Language Modelscited by 40
  8. 2024BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studiescited by 36
  9. 2023GrACE: Generation using Associated Code Editscited by 16
  10. 2024Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Examplecited by 15
  11. 2024GitBug-Java: A Reproducible Benchmark of Recent Java Bugscited by 14
  12. 2019BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixescited by 12
  13. 2023Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editingcited by 9
  14. 2024DyPyBench: A Benchmark of Executable Python Softwarecited by 7
  15. 2023FlaPy: Mining Flaky Python Tests at Scalecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 30.

47 papers.

Sort:

By area:

  1. 2025EnvBench: A Benchmark for Automated Environment Setupcited by 36
  2. 2025Automated Benchmark Generation for Repository-Level Coding Taskscited by 29
  3. 2025Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Computecited by 26
  4. 2025Unified Software Engineering Agent as AI Software Engineercited by 19
  5. 2026SWE-Universe: Scale Real-World Verifiable Environments to Millionscited by 19
  6. 2025Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agentscited by 18
  7. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  8. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 14
  9. 2025Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixingcited by 12
  10. 2026AgentStepper: Interactive Debugging of Software Development Agentscited by 9
  11. 2026EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configurationcited by 8
  12. 2025SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarkscited by 7
35 more
  1. 2026DeployBench: Benchmarking LLM Agents for Research Artifact Deploymentcited by 6
  2. 2026Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skillscited by 5
  3. 2026MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineeringcited by 4
  4. 2026SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environmentscited by 4
  5. 2026DockSmith: Scaling Reliable Coding Environments via an Agentic Docker Buildercited by 3
  6. 2025Treefix: Enabling Execution with a Tree of Prefixescited by 2
  7. 2025LinkAnchor: An Autonomous LLM-Based Agent for Issue-to-Commit Link Recoverycited by 2
  8. 2025Large Language Models for Software Testing: A Research Roadmapcited by 2
  9. 2025Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repaircited by 2
  10. 2026Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agentscited by 2
  11. 2026PaperRepro: Automated Computational Reproducibility Assessment for Social Science Paperscited by 2
  12. 2026Self-Evolving Coding Agentscited by 2
  13. 2025Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulationcited by 1
  14. 2025DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agentscited by 1
  15. 2025Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directionscited by 1
  16. 2026Quality-Driven Agentic Reasoning for LLM-Assisted Software Design: Questions-of-Thoughts (QoT) as a Time-Series Self-QA Chaincited by 1
  17. 2026Early Discoveries of Algorithmist I: Promise of Provable Algorithm Synthesis at Scalecited by 1
  18. 2026Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineeringcited by 1
  19. 2026RAT: RunAnyThing via Fully Automated Environment Configurationcited by 1
  20. 2026Skim: Speculative Execution for Fast and Efficient Web Agentscited by 1
  21. 2026SetupX: Can LLM Agents Learn from Past Failures in Functionality-Correct Code Repository Setup?cited by 1
  22. 2026A Technical Taxonomy of LLM Agent Communication Protocolscited by 1
  23. 2026RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescuecited by 1
  24. 2025SLICEMATE: Accurate and Scalable Static Program Slicing via LLM-Powered Agentscited by 0
  25. 2025PIPer: On-Device Environment Setup via Online Reinforcement Learningcited by 0
  26. 2026Agent-Based Software Artifact Evaluationcited by 0
  27. 2026Evaluating LLM Agents on Automated Software Analysis Taskscited by 0
  28. 2026From Assistance to Agency: Rethinking Autonomy and Control in CI/CD Pipelinescited by 0
  29. 2026BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledgecited by 0
  30. 2026Where Is the Tradeoff in Using Third-Party API Routers for Agentic Software Development?cited by 0
  31. 2026ExplainBench: Evaluating Code Explanations from Agentscited by 0
  32. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  33. 2026Demystifying Agent Skills for Smart Contract Auditing: Design, Effectiveness, Behavioral Impactcited by 0
  34. 2026Graph-Guided Repository Environment Constructioncited by 0
  35. 2026Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired Workflows
  2. 2026Automated structural testing of LLM-based agents: methods, framework, and case studies
  3. 2026Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
  4. 2026Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation
  5. 2026To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair
  6. 2024Agentless: Demystifying LLM-based Software Engineering Agents
  7. 2024The Potential of LLMs in Automating Software Testing: From Generation to Reporting
  8. 2025Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback