# Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback arXiv: https://arxiv.org/abs/2507.18755 ยท HTML: https://mlsanity.com/paper/2507.18755/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 28 of its 49 references that are in the map. - First posted on arXiv 24 Jul 2025. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 28 of 28) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024) - [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024) - [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024) - [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024) - [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025) - [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024) - [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024) - [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025) - [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022) - [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023) - [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025) - [CodeR: Issue Resolving with Multi-Agent and Task Graphs](https://mlsanity.com/paper/2406.01304/) (2024) - [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022) - [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024) - [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024) - [The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason](https://mlsanity.com/paper/2506.12286/) (2025) - [A Systematic Literature Review on Large Language Models for Automated Program Repair](https://mlsanity.com/paper/2405.01466/) (2024) - [Evaluating Agent-based Program Repair at Google](https://mlsanity.com/paper/2501.07531/) (2025) - [FixMiner: Mining Relevant Fix Patterns for Automated Program Repair](https://mlsanity.com/paper/1810.01791/) (2018) - [Trust Enhancement Issues in Program Repair](https://mlsanity.com/paper/2108.13064/) (2021) - [Agentic Bug Reproduction for Effective Automated Program Repair at Google](https://mlsanity.com/paper/2502.01821/) (2025) - [AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation](https://mlsanity.com/paper/2305.12050/) (2023) - [A Survey on Automated Program Repair Techniques](https://mlsanity.com/paper/2303.18184/) (2023) - [Multi-line AI-assisted Code Authoring](https://mlsanity.com/paper/2402.04141/) (2024) - [Neurosymbolic Repair for Low-Code Formula Languages](https://mlsanity.com/paper/2207.11765/) (2022) - [AI-Assisted SQL Authoring at Industry Scale](https://mlsanity.com/paper/2407.13280/) (2024) ## Built on by (most cited first, 6 of 6) - [Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair](https://mlsanity.com/paper/2601.19066/) (2026) - [Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair](https://mlsanity.com/paper/2510.03217/) (2025) - [What Makes a Good Bug Report for an AI Agent?](https://mlsanity.com/paper/2607.07593/) (2026) - [LLM-Based Automated Diagnosis Of Integration Test Failures At Google](https://mlsanity.com/paper/2604.12108/) (2026) - [AuditRepairBench: A Paired-Execution Trace Corpus for Evaluator-Channel Ranking Instability in Agent Repair](https://mlsanity.com/paper/2605.04624/) (2026) - [Reasoning from Traces: Divergence-Guided Agentic Repair of WebAssembly Discrepancies](https://mlsanity.com/paper/2608.05521/) (2026) ## Abstract (arXiv) Aim: With the advent of LLMs, sophisticated agentic program repair has become viable at large organizations with large codebases. In this work, we develop an Engineering Agent that fixes the source code based on test failures at scale across diverse software offerings internally. Method: Using Llama as the base, we employ the ReAct harness to develop an agent. We start with a test failure that was triaged by a rule-based test failure bot. We then set up an agentic harness and allow the agent to reason and run a set of 15 actions from reading a file to generating a patch. We provide feedback to the agent through static analysis and test failures so it can refine its solution. We leverage an LLM-as-a-Judge to ensure that the patch conforms to the standards followed by a human review to land fixes. Benchmark Findings: We curated offline benchmarks for our patch generator, the Engineering Agent loop, and the LLM-as-a-Judge. In offline evaluations we found that a specialized 70B model is highly competitive with the much larger but vanilla Llama-405B. In an ablation study, we found that the ReAct harness (neural model) benefited from the symbolic information from static analysis tools and test execution traces. A model that strikes a balance between the solve rate and error rate vs the cost and latency has a benchmark solve rate of 42.3% using an average 11.8 feedback iterations. Production Findings: In a three month period, 80% of the generated fixes were reviewed, of which 31.5% were landed (25.5% of the total number of generated fixes). Feedback from Engineers: We used open coding to extract qualitative themes from engineers' feedback. We saw positive feedback in the form of quick approvals, gratitude, and surprise. We also found mixed feedback when the Engineering Agent's solution was partially correct and it served as a good starting point.