# PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification

arXiv: https://arxiv.org/abs/2502.02747 · HTML: https://mlsanity.com/paper/2502.02747/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [A Study of Generative Large Language Model for Medical Research and Healthcare](https://mlsanity.com/paper/2305.13523/) (2023)
- [Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model](https://mlsanity.com/paper/2304.13731/) (2023)
- [MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution](https://mlsanity.com/paper/2403.17927/) (2024)
- [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024)
- [Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models](https://mlsanity.com/paper/2406.04271/) (2024)
- [RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph](https://mlsanity.com/paper/2410.14684/) (2024)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [LLM-grounded Video Diffusion Models](https://mlsanity.com/paper/2309.17444/) (2023)
- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement](https://mlsanity.com/paper/2411.00622/) (2024)
- [MASAI: Modular Architecture for Software-engineering AI Agents](https://mlsanity.com/paper/2406.11638/) (2024)
- [Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents](https://mlsanity.com/paper/2408.07060/) (2024)
- [Better patching using LLM prompting, via Self-Consistency](https://mlsanity.com/paper/2306.00108/) (2023)
- [Program Synthesis using Conflict-Driven Learning](https://mlsanity.com/paper/1711.08029/) (2017)

## Built on by (most cited first, 22 of 22)

- [Issue Localization via LLM-Driven Iterative Code Graph Searching](https://mlsanity.com/paper/2503.22424/) (2025)
- [From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs](https://mlsanity.com/paper/2509.01835/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [Co-PatcheR: Collaborative Software Patching with Component(s)-specific Small Reasoning Models](https://mlsanity.com/paper/2505.18955/) (2025)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [TDFlow: Agentic Workflows for Test Driven Development](https://mlsanity.com/paper/2510.23761/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [OpenSage: Self-programming Agent Generation Engine](https://mlsanity.com/paper/2602.16891/) (2026)
- [Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering](https://mlsanity.com/paper/2604.16790/) (2026)
- [HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning](https://mlsanity.com/paper/2511.01104/) (2025)
- [PatchBench: Evaluating AI Agents for Vulnerability Patching](https://mlsanity.com/paper/2609.04075/) (2026)
- [Empowering Autonomous Debugging Agents with Efficient Dynamic Analysis](https://mlsanity.com/paper/2604.24212/) (2026)
- [SoK: Measuring What Matters for Closed-Loop Security Agents](https://mlsanity.com/paper/2510.01654/) (2025)
- [One Tool Is Enough: Reinforcement Learning for Repository-Level LLM Agents](https://mlsanity.com/paper/2512.20957/) (2025)
- [SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning](https://mlsanity.com/paper/2601.21649/) (2026)
- [See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch](https://mlsanity.com/paper/2602.10814/) (2026)
- [Understanding Agent-Based Patching of Compiler Missed Optimizations](https://mlsanity.com/paper/2607.02370/) (2026)
- [Malaika: Understanding Malware through Tri-Grounded Agentic Reasoning](https://mlsanity.com/paper/2607.09179/) (2026)
- [FirmPilot: Evidence-Guided Multi-Agent Environment Recovery for IoT Firmware Rehosting](https://mlsanity.com/paper/2607.14903/) (2026)
- [XAgent: eXecution-guided Agentic AI for Effective Localization and Resolution of GitHub Issues](https://mlsanity.com/paper/2609.09769/) (2026)
- [AdaRepair-Mem: Adaptive Experience Orchestration for Repository-Level Program Repair](https://mlsanity.com/paper/2609.20130/) (2026)
- [SelfOp: An Optimization Algorithm for Self-Improving Security Agents](https://mlsanity.com/paper/2609.22792/) (2026)

## Abstract (arXiv)

Recent research builds various patching agents that combine large language models (LLMs) with non-ML tools and achieve promising results on the state-of-the-art (SOTA) software patching benchmark, SWE-bench. Based on how to determine the patching workflows, existing patching agents can be categorized as agent-based planning methods, which rely on LLMs for planning, and rule-based planning methods, which follow a pre-defined workflow. At a high level, agent-based planning methods achieve high patching performance but with a high cost and limited stability. Rule-based planning methods, on the other hand, are more stable and efficient but have key workflow limitations that compromise their patching performance. In this paper, we propose PatchPilot, an agentic patcher that strikes a balance between patching efficacy, stability, and cost-efficiency. PatchPilot proposes a novel rule-based planning workflow with five components: reproduction, localization, generation, validation, and refinement (where refinement is unique to PatchPilot). We introduce novel and customized designs to each component to optimize their effectiveness and efficiency. Through extensive experiments on the SWE-bench benchmarks, PatchPilot shows a superior performance than existing open-source methods while maintaining low cost (less than 1 per instance) and ensuring higher stability. We also conduct a detailed ablation study to validate the key designs in each component. Our code is available at https://github.com/ucsb-mlsec/PatchPilot.