# Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair

arXiv: https://arxiv.org/abs/2609.00854 · HTML: https://mlsanity.com/paper/2609.00854/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Automated Repair of Programs from Large Language Models](https://mlsanity.com/paper/2205.10583/) (2022)
- [RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair](https://mlsanity.com/paper/2312.15698/) (2023)
- [Aligning the Objective of LLM-based Program Repair](https://mlsanity.com/paper/2404.08877/) (2024)
- [Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges](https://mlsanity.com/paper/2506.04418/) (2025)
- [RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Models](https://mlsanity.com/paper/2601.18044/) (2026)
- [Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models](https://mlsanity.com/paper/2606.31511/) (2026)
- [On the Role of Fault Localization Context for LLM-Based Program Repair](https://mlsanity.com/paper/2604.05481/) (2026)
- [Loc2Repair: A Framework for Evaluating the Impact of File-Level Issue Localization in Repo-Level LLM Repair](https://mlsanity.com/paper/2606.30963/) (2026)
- [Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems](https://mlsanity.com/paper/2608.25920/) (2026)
- [Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?](https://mlsanity.com/paper/2604.17338/) (2026)
- [SHERLOC: Structured Diagnostic Localization for Code Repair Agents](https://mlsanity.com/paper/2606.24820/) (2026)
- [Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models](https://mlsanity.com/paper/2607.12962/) (2026)
- [Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models](https://mlsanity.com/paper/2607.26117/) (2026)
- [A Study on the Impact of Fault localization Granularity for Repository-Scale Code Repair Tasks](https://mlsanity.com/paper/2604.00167/) (2026)
- [Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency](https://mlsanity.com/paper/2608.14065/) (2026)

## Abstract (arXiv)

Fault localization can focus a code model's repair on the statements a failing test implicates, but a targeted edit may succeed merely because it is small, and a second model call may succeed without using the failure at all. We separate these explanations with three arms applied to the same failed candidate: blind whole-solution resampling, spectrum-based localization followed by suspect-span infilling, and same-length infilling at a disjoint random code span. Across three frozen 26-32B models, three benchmarks and 488 failing candidates, plus a separately declared 24B fourth model from a third family, three results follow. First, localization is rarely available: only 9.0% of failing candidates expose a failing public test with a usable spectrum. Second, among the 177 candidates localizable from a strong suite, localized infilling loses decisively to blind resampling at a matched attempt count (3:40, p = 3.0 x 10^-9), opposite to our hypothesis; the loss replicates in a third family at -11.3 points (95% CI [-16.6, -6.8]), and widening the edit does not rescue it. Third, against the random-span placebo localized infilling leads pooled (11:1, Holm-adjusted p = .019), but that lead resolves in no individual model under the analysis our shipped plan designates primary (best Holm p = .087), so we report the location effect as suggestive rather than established. Re-pricing attempts as tokens narrows but does not overturn this: a span attempt spends 21.7 generated tokens against 371.1, yet 16 localized attempts reach 6.8% while one blind attempt already reaches 10.1%. Infilling reproduces the removed span verbatim in 48.9% of attempts, which is why more budget does not help. We restrict every localization conclusion to the 24-32B models tested.