# Large Language Models for Test-Free Fault Localization

arXiv: https://arxiv.org/abs/2310.01726 · HTML: https://mlsanity.com/paper/2310.01726/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Program Synthesis using Natural Language](https://mlsanity.com/paper/1509.00413/) (2015)
- [On the "Naturalness" of Buggy Code](https://mlsanity.com/paper/1506.01159/) (2015)
- [DEAR: A Novel Deep Learning-based Approach for Automated Program Repair](https://mlsanity.com/paper/2205.01859/) (2022)
- [Fault Localization with Code Coverage Representation Learning](https://mlsanity.com/paper/2103.00270/) (2021)
- [Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2,141 Bugs and 23,551 Repair Attempts](https://mlsanity.com/paper/1905.11973/) (2019)
- [A Large-Scale Study of Call Graph-based Impact Prediction using Mutation Testing](https://mlsanity.com/paper/1812.06286/) (2018)
- [Assessing and Comparing Mutation-based Fault Localization Techniques](https://mlsanity.com/paper/1607.05512/) (2016)

## Built on by (most cited first, 30 of 67)

- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution](https://mlsanity.com/paper/2507.23348/) (2025)
- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [OrcaLoca: An LLM Agent Framework for Software Issue Localization](https://mlsanity.com/paper/2502.00350/) (2025)
- [UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debugging](https://mlsanity.com/paper/2404.17153/) (2024)
- [Evaluating Agent-based Program Repair at Google](https://mlsanity.com/paper/2501.07531/) (2025)
- [How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models](https://mlsanity.com/paper/2404.18353/) (2024)
- [Agentic Bug Reproduction for Effective Automated Program Repair at Google](https://mlsanity.com/paper/2502.01821/) (2025)
- [SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs](https://mlsanity.com/paper/2504.14757/) (2025)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [Security Vulnerability Detection with Multitask Self-Instructed Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2406.05892/) (2024)
- [Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub](https://mlsanity.com/paper/2601.15195/) (2026)
- [Are Large Language Models Memorizing Bug Benchmarks?](https://mlsanity.com/paper/2411.13323/) (2024)
- [LLMs: A Game-Changer for Software Engineers?](https://mlsanity.com/paper/2411.00932/) (2024)
- [VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection](https://mlsanity.com/paper/2512.07533/) (2025)
- [Location is Key: Leveraging Large Language Model for Functional Bug Localization in Verilog](https://mlsanity.com/paper/2409.15186/) (2024)
- [Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines](https://mlsanity.com/paper/2510.07614/) (2025)
- [ChatDBG: Augmenting Debugging with Large Language Models](https://mlsanity.com/paper/2403.16354/) (2024)
- [A Multi-Agent Approach to Fault Localization via Graph-Based Retrieval and Reflexion](https://mlsanity.com/paper/2409.13642/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [BLAZE: Cross-Language and Cross-Project Bug Localization via Dynamic Chunking and Hard Example Learning](https://mlsanity.com/paper/2407.17631/) (2024)
- [SuperCoder: Assembly Program Superoptimization with Large Language Models](https://mlsanity.com/paper/2505.11480/) (2025)
- [One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discovery](https://mlsanity.com/paper/2510.14036/) (2025)
- [RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair](https://mlsanity.com/paper/2603.01048/) (2026)
- [MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals](https://mlsanity.com/paper/2605.07509/) (2026)
- [EmbedAgent: Benchmarking Large Language Models in Embedded System Development](https://mlsanity.com/paper/2506.11003/) (2025)

## Abstract (arXiv)

Fault Localization (FL) aims to automatically localize buggy lines of code, a key first step in many manual and automatic debugging tasks. Previous FL techniques assume the provision of input tests, and often require extensive program analysis, program instrumentation, or data preprocessing. Prior work on deep learning for APR struggles to learn from small datasets and produces limited results on real-world programs. Inspired by the ability of large language models (LLMs) of code to adapt to new tasks based on very few examples, we investigate the applicability of LLMs to line level fault localization. Specifically, we propose to overcome the left-to-right nature of LLMs by fine-tuning a small set of bidirectional adapter layers on top of the representations learned by LLMs to produce LLMAO, the first language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune LLMs with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs such as the Defects4J corpus. We observe that our technique achieves substantially more confidence in fault localization when built on the larger models, with bug localization performance scaling consistently with the LLM size. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%-54.4%, and Top-5 results by 14.4%-35.6%. LLMAO is also the first FL technique trained using a language model architecture that can detect security vulnerabilities down to the code line level.