mlsanityData to 14 Oct 2026

Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair

arXiv 2609.00854 ↗First posted 1 Sep 2026cs.SE cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Fault localization can focus a code model's repair on the statements a failing test implicates, but a targeted edit may succeed merely because it is small, and a second model call may succeed without using the failure at all. We separate these explanations with three arms applied to the same failed candidate: blind whole-solution resampling, spectrum-based localization followed by suspect-span infilling, and same-length infilling at a disjoint random code span. Across three frozen 26-32B models, three benchmarks and 488 failing candidates, plus a separately declared 24B fourth model from a third family, three results follow. First, localization is rarely available: only 9.0% of failing candidates expose a failing public test with a usable spectrum. Second, among the 177 candidates localizable from a strong suite, localized infilling loses decisively to blind resampling at a matched attempt count (3:40, p = 3.0 x 10^-9), opposite to our hypothesis; the loss replicates in a third family at -11.3 points (95% CI [-16.6, -6.8]), and widening the edit does not rescue it. Third, against the random-span placebo localized infilling leads pooled (11:1, Holm-adjusted p = .019), but that lead resolves in no individual model under the analysis our shipped plan designates primary (best Holm p = .087), so we report the location effect as suggestive rather than established. Re-pricing attempts as tokens narrows but does not overturn this: a span attempt spends 21.7 generated tokens against 371.1, yet 16 localized attempts reach 6.8% while one blind attempt already reaches 10.1%. Infilling reproduces the removed span verbatim in 48.9% of attempts, which is why more budget does not help. We restrict every localization conclusion to the 24-32B models tested.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 20 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

17 of 20 references.

References · 17

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  3. 2022Automated Repair of Programs from Large Language Modelscited by 71
  4. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  5. 2024Aligning the Objective of LLM-based Program Repaircited by 12
  6. 2025Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challengescited by 7
  7. 2026RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Modelscited by 5
  8. 2026Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Modelscited by 4
  9. 2026On the Role of Fault Localization Context for LLM-Based Program Repaircited by 3
  10. 2026Loc2Repair: A Framework for Evaluating the Impact of File-Level Issue Localization in Repo-Level LLM Repaircited by 3
  11. 2026Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systemscited by 3
  12. 2026Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?cited by 2
5 more
  1. 2026SHERLOC: Structured Diagnostic Localization for Code Repair Agentscited by 2
  2. 2026Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Modelscited by 2
  3. 2026Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Modelscited by 2
  4. 2026A Study on the Impact of Fault localization Granularity for Repository-Scale Code Repair Taskscited by 1
  5. 2026Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiencycited by 1

Built on by

Not cited yet.