mlsanityData to 7 Oct 2026

An Empirical Study on LLM-based Agents for Automated Bug Fixing

arXiv 2411.10213 ↗First posted 15 Nov 2024cs.SE cs.AIcited by 23

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Large language models (LLMs) and LLM-based Agents have been applied to fix bugs automatically, demonstrating the capability in addressing software defects by engaging in development environment interaction, iterative validation and code modification. However, systematic analysis of these agent systems remain limited, particularly regarding performance variations among top-performing ones. In this paper, we examine six repair systems on the SWE-bench Verified benchmark for automated bug fixing. We first assess each system's overall performance, noting the instances solvable by all or none of these systems, and explore the capabilities of different systems. We also compare fault localization accuracy at file and code symbol levels and evaluate bug reproduction capabilities. Through analysis, we concluded that further optimization is needed in both the LLM capability itself and the design of Agentic flow to improve the effectiveness of the Agent in bug fixing.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 63 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

20 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement26 Oct
  2. 2024MarsCode Agent: AI-native Automated Bug Fixing2 Sep
  3. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution26 Mar

Other references · 17

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  6. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  7. 2023Automatic Root Cause Analysis via Large Language Models for Cloud Incidentscited by 82
  8. 2023Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learningcited by 71
  9. 2024Exploring and Unleashing the Power of Large Language Models in Automated Code Translationcited by 57
  10. 2023A Survey of Learning-based Automated Program Repaircited by 55
  11. 2024AgentFL: Scaling LLM-based Fault Localization to Project-Level Contextcited by 47
  12. 2023Explainable Automated Debugging via Large Language Model-driven Scientific Debuggingcited by 40
5 more
  1. 2024Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuningcited by 30
  2. 2022DEAR: A Novel Deep Learning-based Approach for Automated Program Repaircited by 22
  3. 2023CCT5: A Code-Change-Oriented Pre-Trained Modelcited by 17
  4. 2024Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketchescited by 7
  5. 2023Enhancing Code Intelligence Tasks with ChatGPTcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025Issue Localization via LLM-Driven Iterative Code Graph Searchingcited by 22
  2. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  3. 2025cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Treecited by 17
  4. 2025Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixingcited by 11
  5. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failurecited by 9
  6. 2025Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systemscited by 8
  7. 2025A Comprehensive Empirical Evaluation of Agent Frameworks on Code-centric Software Engineering Taskscited by 7
  8. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  9. 2025Scalable Supervision for Software Agents via Patch Reasoningcited by 5
  10. 2026RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Modelscited by 5
  11. 2024Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repaircited by 4
  12. 2025Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilitiescited by 4
11 more
  1. 2026CodeScout: Contextual Problem Statement Enhancement for Software Agentscited by 4
  2. 2026REAgent: Requirement-Driven LLM Agents for Software Issue Resolutioncited by 4
  3. 2025Empirical Research on Utilizing LLM-based Agents for Automated Bug Fixing via LangGraphcited by 3
  4. 2025HAFixAgent: History-Aware Program Repair Agentcited by 2
  5. 2025From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineeringcited by 2
  6. 2026VulnResolver: A Hybrid Agent Framework for LLM-Based Automated Vulnerability Issue Resolutioncited by 2
  7. 2026Towards Structured, State-Aware, and Execution-Grounded Reasoning for Software Engineering Agentscited by 2
  8. 2026ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agentscited by 1
  9. 2026A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedbackcited by 1
  10. 2026TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenancecited by 0
  11. 2026Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents
  2. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
  3. 2026Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair
  4. 2025Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
  5. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs
  6. 2025Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback
  7. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
  8. 2026Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency