# Large Language Models in Fault Localisation

arXiv: https://arxiv.org/abs/2308.15276 · HTML: https://mlsanity.com/paper/2308.15276/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2023.

## Built on by (most cited first, 28 of 28)

- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [LocAgent: Graph-Guided LLM Agents for Code Localization](https://mlsanity.com/paper/2503.09089/) (2025)
- [SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution](https://mlsanity.com/paper/2507.23348/) (2025)
- [OrcaLoca: An LLM Agent Framework for Software Issue Localization](https://mlsanity.com/paper/2502.00350/) (2025)
- [Issue Localization via LLM-Driven Iterative Code Graph Searching](https://mlsanity.com/paper/2503.22424/) (2025)
- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [Feedback-Generation for Programming Exercises With GPT-4](https://mlsanity.com/paper/2403.04449/) (2024)
- [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023)
- [A Multi-Agent Approach to Fault Localization via Graph-Based Retrieval and Reflexion](https://mlsanity.com/paper/2409.13642/) (2024)
- [Evaluating Large Language Models for Real-World Engineering Tasks](https://mlsanity.com/paper/2505.13484/) (2025)
- [DCA-Bench: A Benchmark for Dataset Curation Agents](https://mlsanity.com/paper/2406.07275/) (2024)
- [You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?](https://mlsanity.com/paper/2412.03516/) (2024)
- [LUNA: A Model-Based Universal Analysis Framework for Large Language Models](https://mlsanity.com/paper/2310.14211/) (2023)
- [Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization](https://mlsanity.com/paper/2412.18750/) (2024)
- [COSMosFL: Ensemble of Small Language Models for Fault Localisation](https://mlsanity.com/paper/2502.02908/) (2025)
- [Where's the Bug? Attention Probing for Scalable Fault Localization](https://mlsanity.com/paper/2502.13966/) (2025)
- [PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents](https://mlsanity.com/paper/2607.18859/) (2026)
- [Fault Localization via Fine-tuning Large Language Models with Mutation Generated Stack Traces](https://mlsanity.com/paper/2501.18005/) (2025)
- [Towards Adaptive Software Agents for Debugging](https://mlsanity.com/paper/2504.18316/) (2025)
- [Root Cause Analysis Method Based on Large Language Models with Residual Connection Structures](https://mlsanity.com/paper/2602.08804/) (2026)
- [Multi-task LLMs for Bug Classification: Efficient Inference with Auxiliary Decoding Heads](https://mlsanity.com/paper/2606.09956/) (2026)
- [Breaking the Silence: the Threats of Using LLMs in Software Engineering](https://mlsanity.com/paper/2312.08055/) (2023)
- [Demystifying Faulty Code with LLM: Step-by-Step Reasoning for Explainable Fault Localization](https://mlsanity.com/paper/2403.10507/) (2024)
- [Inferring Pluggable Types with Machine Learning](https://mlsanity.com/paper/2406.15676/) (2024)
- [Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmap](https://mlsanity.com/paper/2505.19625/) (2025)
- [CodeFlowLM: Incremental Just-In-Time Defect Prediction with Pretrained Language Models and Exploratory Insights into Defect Localization](https://mlsanity.com/paper/2512.00231/) (2025)
- [LLM-Based Automated Diagnosis Of Integration Test Failures At Google](https://mlsanity.com/paper/2604.12108/) (2026)
- [BLAgent: Agentic RAG for File-Level Bug Localization](https://mlsanity.com/paper/2605.17965/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promise in multiple software engineering tasks including code generation, program repair, code summarisation, and test generation. Fault localisation is instrumental in enabling automated debugging and repair of programs and was prominently featured as a highlight during the launch event of ChatGPT-4. Nevertheless, the performance of LLMs compared to state-of-the-art methods, as well as the impact of prompt design and context length on their efficacy, remains unclear. To fill this gap, this paper presents an in-depth investigation into the capability of ChatGPT-3.5 and ChatGPT-4, the two state-of-the-art LLMs, on fault localisation. Using the widely-adopted large-scale Defects4J dataset, we compare the two LLMs with the existing fault localisation techniques. We also investigate the consistency of LLMs in fault localisation, as well as how prompt engineering and the length of code context affect the fault localisation effectiveness. Our findings demonstrate that within function-level context, ChatGPT-4 outperforms all the existing fault localisation methods. Additional error logs can further improve ChatGPT models' localisation accuracy and consistency, with an average 46.9% higher accuracy over the state-of-the-art baseline SmartFL on the Defects4J dataset in terms of TOP-1 metric. However, when the code context of the Defects4J dataset expands to the class-level, ChatGPT-4's performance suffers a significant drop, with 49.9% lower accuracy than SmartFL under TOP-1 metric. These observations indicate that although ChatGPT can effectively localise faults under specific conditions, limitations are evident. Further research is needed to fully harness the potential of LLMs like ChatGPT for practical fault localisation applications.