# The Fact Selection Problem in LLM-Based Program Repair

arXiv: https://arxiv.org/abs/2404.05520 · HTML: https://mlsanity.com/paper/2404.05520/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2024.

## Built on by (most cited first, 12 of 12)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Statically Contextualizing Large Language Models with Typed Holes](https://mlsanity.com/paper/2409.00921/) (2024)
- [RepairBench: Leaderboard of Frontier Models for Program Repair](https://mlsanity.com/paper/2409.18952/) (2024)
- [DocMEdit: Towards Document-Level Model Editing](https://mlsanity.com/paper/2505.19572/) (2025)
- [Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repair](https://mlsanity.com/paper/2412.03905/) (2024)
- [Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects](https://mlsanity.com/paper/2502.17441/) (2024)
- [CONCUR: Benchmarking LLMs for Concurrent Code Generation](https://mlsanity.com/paper/2603.03683/) (2026)
- [Design-Specification Tiling for ICL-based CAD Code Generation](https://mlsanity.com/paper/2603.12712/) (2026)
- [How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair](https://mlsanity.com/paper/2607.25873/) (2026)
- [When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences](https://mlsanity.com/paper/2607.28384/) (2026)
- [Recompilation Is Not Enough: Test-Guided Decompiled-C Repair](https://mlsanity.com/paper/2609.07201/) (2026)
- [CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verification](https://mlsanity.com/paper/2609.07434/) (2026)

## Abstract (arXiv)

Recent research has shown that incorporating bug-related facts, such as stack traces and GitHub issues, into prompts enhances the bug-fixing capabilities of large language models (LLMs). Considering the ever-increasing context window of these models, a critical question arises: what and how many facts should be included in prompts to maximise the chance of correctly fixing bugs? To answer this question, we conducted a large-scale study, employing over 19K prompts featuring various combinations of seven diverse facts to rectify 314 bugs from open-source Python projects within the BugsInPy benchmark. Our findings revealed that each fact, ranging from simple syntactic details like code context to semantic information previously unexplored in the context of LLMs such as angelic values, is beneficial. Specifically, each fact aids in fixing some bugs that would remain unresolved or only be fixed with a low success rate without it. Importantly, we discovered that the effectiveness of program repair prompts is non-monotonic over the number of used facts; using too many facts leads to subpar outcomes. These insights led us to define the fact selection problem: determining the optimal set of facts for inclusion in a prompt to maximise LLM's performance on a given task instance. We found that there is no one-size-fits-all set of facts for bug repair. Therefore, we developed a basic statistical model, named Maniple, which selects facts specific to a given bug to include in the prompt. This model significantly surpasses the performance of the best generic fact set. To underscore the significance of the fact selection problem, we benchmarked Maniple against the state-of-the-art zero-shot, non-conversational LLM-based bug repair methods. On our testing dataset of 157 bugs, Maniple repairs 88 bugs, 17% above the best configuration.