# An Empirical Study on LLM-based Agents for Automated Bug Fixing

arXiv: https://arxiv.org/abs/2411.10213 · HTML: https://mlsanity.com/paper/2411.10213/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 63 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution](https://mlsanity.com/paper/2403.17927/) (2024)
- [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024)
- [Automatic Root Cause Analysis via Large Language Models for Cloud Incidents](https://mlsanity.com/paper/2305.15778/) (2023)
- [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023)
- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [Exploring and Unleashing the Power of Large Language Models in Automated Code Translation](https://mlsanity.com/paper/2404.14646/) (2024)
- [A Survey of Learning-based Automated Program Repair](https://mlsanity.com/paper/2301.03270/) (2023)
- [AgentFL: Scaling LLM-based Fault Localization to Project-Level Context](https://mlsanity.com/paper/2403.16362/) (2024)
- [Explainable Automated Debugging via Large Language Model-driven Scientific Debugging](https://mlsanity.com/paper/2304.02195/) (2023)
- [Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning](https://mlsanity.com/paper/2406.03718/) (2024)
- [DEAR: A Novel Deep Learning-based Approach for Automated Program Repair](https://mlsanity.com/paper/2205.01859/) (2022)
- [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023)
- [Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketches](https://mlsanity.com/paper/2405.03998/) (2024)
- [Enhancing Code Intelligence Tasks with ChatGPT](https://mlsanity.com/paper/2312.15202/) (2023)

## Built on by (most cited first, 23 of 23)

- [Issue Localization via LLM-Driven Iterative Code Graph Searching](https://mlsanity.com/paper/2503.22424/) (2025)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree](https://mlsanity.com/paper/2506.15655/) (2025)
- [Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing](https://mlsanity.com/paper/2506.16136/) (2025)
- [Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure](https://mlsanity.com/paper/2604.02547/) (2026)
- [Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems](https://mlsanity.com/paper/2506.17208/) (2025)
- [A Comprehensive Empirical Evaluation of Agent Frameworks on Code-centric Software Engineering Tasks](https://mlsanity.com/paper/2511.00872/) (2025)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities](https://mlsanity.com/paper/2509.25894/) (2025)
- [Scalable Supervision for Software Agents via Patch Reasoning](https://mlsanity.com/paper/2510.22775/) (2025)
- [RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Models](https://mlsanity.com/paper/2601.18044/) (2026)
- [Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repair](https://mlsanity.com/paper/2412.03905/) (2024)
- [CodeScout: Contextual Problem Statement Enhancement for Software Agents](https://mlsanity.com/paper/2603.05744/) (2026)
- [REAgent: Requirement-Driven LLM Agents for Software Issue Resolution](https://mlsanity.com/paper/2604.06861/) (2026)
- [Empirical Research on Utilizing LLM-based Agents for Automated Bug Fixing via LangGraph](https://mlsanity.com/paper/2502.18465/) (2025)
- [HAFixAgent: History-Aware Program Repair Agent](https://mlsanity.com/paper/2511.01047/) (2025)
- [From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering](https://mlsanity.com/paper/2512.23844/) (2025)
- [VulnResolver: A Hybrid Agent Framework for LLM-Based Automated Vulnerability Issue Resolution](https://mlsanity.com/paper/2601.13933/) (2026)
- [Towards Structured, State-Aware, and Execution-Grounded Reasoning for Software Engineering Agents](https://mlsanity.com/paper/2602.04640/) (2026)
- [ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents](https://mlsanity.com/paper/2604.07789/) (2026)
- [A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback](https://mlsanity.com/paper/2605.18073/) (2026)
- [TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance](https://mlsanity.com/paper/2601.18241/) (2026)
- [Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment](https://mlsanity.com/paper/2606.24834/) (2026)

## Abstract (arXiv)

Large language models (LLMs) and LLM-based Agents have been applied to fix bugs automatically, demonstrating the capability in addressing software defects by engaging in development environment interaction, iterative validation and code modification. However, systematic analysis of these agent systems remain limited, particularly regarding performance variations among top-performing ones. In this paper, we examine six repair systems on the SWE-bench Verified benchmark for automated bug fixing. We first assess each system's overall performance, noting the instances solvable by all or none of these systems, and explore the capabilities of different systems. We also compare fault localization accuracy at file and code symbol levels and evaluate bug reproduction capabilities. Through analysis, we concluded that further optimization is needed in both the LLM capability itself and the design of Agentic flow to improve the effectiveness of the Agent in bug fixing.