# How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

arXiv: https://arxiv.org/abs/2607.25873 · HTML: https://mlsanity.com/paper/2607.25873/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 83 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 30 of 42)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025)
- [Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs](https://mlsanity.com/paper/2311.02262/) (2023)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [On the Role of Attention Heads in Large Language Model Safety](https://mlsanity.com/paper/2410.13708/) (2024)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarization](https://mlsanity.com/paper/2402.14182/) (2024)
- [Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?](https://mlsanity.com/paper/2306.01220/) (2023)
- [You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems](https://mlsanity.com/paper/1812.07283/) (2018)
- [Assessing the Quality of the Steps to Reproduce in Bug Reports](https://mlsanity.com/paper/1906.07107/) (2019)
- [Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation](https://mlsanity.com/paper/2603.06865/) (2026)
- [Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Training](https://mlsanity.com/paper/2009.12064/) (2020)
- [Insights into Natural Language Database Query Errors: From Attention Misalignment to User Handling Strategies](https://mlsanity.com/paper/2402.07304/) (2024)
- [The Fact Selection Problem in LLM-Based Program Repair](https://mlsanity.com/paper/2404.05520/) (2024)
- [Attention Meets Post-hoc Interpretability: A Mathematical Perspective](https://mlsanity.com/paper/2402.03485/) (2024)
- [Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution](https://mlsanity.com/paper/2501.11709/) (2025)
- [Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs](https://mlsanity.com/paper/2505.04441/) (2025)

## Abstract (arXiv)

Large Language Model (LLM)-based Automated Program Repair systems are advancing rapidly, yet their performance remains inconsistent. Even when provided with the same contextual information, an LLM may generate a correct patch for one bug but fail on another closely related bug. Why this happens remains poorly understood, and it is unclear how LLMs prioritize the diverse information in bug reports and whether model attention affects repair success. In this paper, we present the first empirical study of attention patterns in LLM-based program repair, providing interpretable insights into how models process bug reports and where their attention is concentrated during repair. We analyze 319 real-world Python and Java bugs from SWE-bench Verified and Multi-SWE-bench to study (RQ1) how model attention is distributed across bug report sections, (RQ2) how attention patterns within each section differ between successful and unsuccessful repairs, and (RQ3) how these patterns compare to information developers consider important for bug fixing. We find that successful repairs are characterized by diffused attention across multiple diagnostic components such as bug descriptions, stacktraces, and test cases, while failures often exhibit over-localized attention toward metadata such as version information. We further observe that stronger alignment between model attention and developer-identified key sections and phrases is associated with higher repair success. Our results provide the first empirical evidence that attention misallocation is a key factor in LLM-based APR failures, and offer actionable insights for designing more interpretable and reliable future APR systems.