# Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair

arXiv: https://arxiv.org/abs/2609.04909 · HTML: https://mlsanity.com/paper/2609.04909/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 61 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [When Agents go Astray: Course-Correcting SWE Agents with PRMs](https://mlsanity.com/paper/2509.02360/) (2025)
- [Agentic Bug Reproduction for Effective Automated Program Repair at Google](https://mlsanity.com/paper/2502.01821/) (2025)
- [De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding](https://mlsanity.com/paper/2401.01701/) (2024)
- [Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges](https://mlsanity.com/paper/2504.20799/) (2025)
- [RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair](https://mlsanity.com/paper/2603.01048/) (2026)
- [From Guessing to Seeing: Enhancing LLM-Based Program Repair via Trace-Guided Multi-strategy Debate](https://mlsanity.com/paper/2604.02647/) (2026)
- [Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities](https://mlsanity.com/paper/2509.25894/) (2025)
- [When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?](https://mlsanity.com/paper/2510.17862/) (2025)
- [HAFixAgent: History-Aware Program Repair Agent](https://mlsanity.com/paper/2511.01047/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation. We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior.