# A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization

arXiv: https://arxiv.org/abs/2308.05487 · HTML: https://mlsanity.com/paper/2308.05487/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2023.

## Built on by (most cited first, 30 of 50)

- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [LocAgent: Graph-Guided LLM Agents for Code Localization](https://mlsanity.com/paper/2503.09089/) (2025)
- [HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale](https://mlsanity.com/paper/2409.16299/) (2024)
- [Issue Localization via LLM-Driven Iterative Code Graph Searching](https://mlsanity.com/paper/2503.22424/) (2025)
- [Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects](https://mlsanity.com/paper/2412.06294/) (2024)
- [Migrating Code At Scale With LLMs At Google](https://mlsanity.com/paper/2504.09691/) (2025)
- [Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub](https://mlsanity.com/paper/2601.15195/) (2026)
- [ChatDBG: Augmenting Debugging with Large Language Models](https://mlsanity.com/paper/2403.16354/) (2024)
- [A Multi-Agent Approach to Fault Localization via Graph-Based Retrieval and Reflexion](https://mlsanity.com/paper/2409.13642/) (2024)
- [Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy](https://mlsanity.com/paper/2503.00481/) (2025)
- [Retrieval-Augmented Test Generation: How Far Are We?](https://mlsanity.com/paper/2409.12682/) (2024)
- [Enhancing Automated Loop Invariant Generation for Complex Programs with Large Language Models](https://mlsanity.com/paper/2412.10483/) (2024)
- [On the Effectiveness of Large Language Models in Writing Alloy Formulas](https://mlsanity.com/paper/2502.15441/) (2025)
- [MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals](https://mlsanity.com/paper/2605.07509/) (2026)
- [GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis?](https://mlsanity.com/paper/2508.12472/) (2025)
- [RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Models](https://mlsanity.com/paper/2601.18044/) (2026)
- [Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repair](https://mlsanity.com/paper/2412.03905/) (2024)
- [Assessing the Impact of Code Changes on the Fault Localizability of Large Language Models](https://mlsanity.com/paper/2504.04372/) (2025)
- [From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization](https://mlsanity.com/paper/2510.02389/) (2025)
- [Past, Present, and Future of Bug Tracking in the Generative AI Era](https://mlsanity.com/paper/2510.08005/) (2025)
- [Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization](https://mlsanity.com/paper/2412.18750/) (2024)
- [COSMosFL: Ensemble of Small Language Models for Fault Localisation](https://mlsanity.com/paper/2502.02908/) (2025)
- [Localized Calibrated Uncertainty in Code Language Models](https://mlsanity.com/paper/2512.24560/) (2025)
- [Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap](https://mlsanity.com/paper/2604.15075/) (2026)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities](https://mlsanity.com/paper/2409.10574/) (2024)
- [LinkAnchor: An Autonomous LLM-Based Agent for Issue-to-Commit Link Recovery](https://mlsanity.com/paper/2508.12232/) (2025)
- [OBsmith: LLM-Powered JavaScript Obfuscator Testing](https://mlsanity.com/paper/2510.10066/) (2025)
- [SHERLOC: Structured Diagnostic Localization for Code Repair Agents](https://mlsanity.com/paper/2606.24820/) (2026)
- [PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents](https://mlsanity.com/paper/2607.18859/) (2026)

## Abstract (arXiv)

Fault Localization (FL), in which a developer seeks to identify which part of the code is malfunctioning and needs to be fixed, is a recurring challenge in debugging. To reduce developer burden, many automated FL techniques have been proposed. However, prior work has noted that existing techniques fail to provide rationales for the suggested locations, hindering developer adoption of these techniques. With this in mind, we propose AutoFL, a Large Language Model (LLM)-based FL technique that generates an explanation of the bug along with a suggested fault location. AutoFL prompts an LLM to use function calls to navigate a repository, so that it can effectively localize faults over a large software repository and overcome the limit of the LLM context length. Extensive experiments on 798 real-world bugs in Java and Python reveal AutoFL improves method-level acc@1 by up to 233.3% over baselines. Furthermore, developers were interviewed on their impression of AutoFL-generated explanations, showing that developers generally liked the natural language explanations of AutoFL, and that they preferred reading a few, high-quality explanations instead of many.