mlsanityData to 7 Oct 2026

How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

arXiv 2607.25873 ↗First posted 28 Jul 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM)-based Automated Program Repair systems are advancing rapidly, yet their performance remains inconsistent. Even when provided with the same contextual information, an LLM may generate a correct patch for one bug but fail on another closely related bug. Why this happens remains poorly understood, and it is unclear how LLMs prioritize the diverse information in bug reports and whether model attention affects repair success. In this paper, we present the first empirical study of attention patterns in LLM-based program repair, providing interpretable insights into how models process bug reports and where their attention is concentrated during repair. We analyze 319 real-world Python and Java bugs from SWE-bench Verified and Multi-SWE-bench to study (RQ1) how model attention is distributed across bug report sections, (RQ2) how attention patterns within each section differ between successful and unsuccessful repairs, and (RQ3) how these patterns compare to information developers consider important for bug fixing. We find that successful repairs are characterized by diffused attention across multiple diagnostic components such as bug descriptions, stacktraces, and test cases, while failures often exhibit over-localized attention toward metadata such as version information. We further observe that stronger alignment between model attention and developer-identified key sections and phrases is associated with higher repair success. Our results provide the first empirical evidence that attention misallocation is a key factor in LLM-based APR failures, and offer actionable insights for designing more interpretable and reliable future APR systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 83 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

42 of 83 references.

References · 42

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  4. 2019Attention is not Explanationcited by 916
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  7. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  8. 2019Revealing the Dark Secrets of BERTcited by 300
  9. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  10. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  11. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  12. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
30 more
  1. 2023InferFix: End-to-End Program Repair with LLMscited by 113
  2. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 91
  3. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  4. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  5. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  6. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  7. 2024Attention Heads of Large Language Models: A Surveycited by 65
  8. 2024Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarizationcited by 16
  9. 2018You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systemscited by 13
  10. 2019Assessing the Quality of the Steps to Reproduce in Bug Reportscited by 13
  11. 2023Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?cited by 13
  12. 2020Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Trainingcited by 12
  13. 2024The Fact Selection Problem in LLM-Based Program Repaircited by 12
  14. 2024Attention Meets Post-hoc Interpretability: A Mathematical Perspectivecited by 11
  15. 2024Insights into Natural Language Database Query Errors: From Attention Misalignment to User Handling Strategiescited by 11
  16. 2025Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolutioncited by 11
  17. 2025Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMscited by 11
  18. 2026Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluationcited by 11
  19. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 8
  20. 2024Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'cited by 7
  21. 2025Hierarchical Knowledge Injection for Improving LLM-based Program Repaircited by 6
  22. 2025Incoherence as Oracle-less Measure of Error in LLM-Based Code Generationcited by 6
  23. 2022Follow-up Attention: An Empirical Study of Developer and Neural Model Code Explorationcited by 5
  24. 2024Memory-Efficient Large Language Models for Program Repair with Semantic-Guided Patch Generationcited by 5
  25. 2025What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issuescited by 5
  26. 2022BugListener: Identifying and Synthesizing Bug Reports from Collaborative Live Chatscited by 4
  27. 2025HAFix: History-Augmented Large Language Models for Bug Fixingcited by 4
  28. 2025PSearch: Search-based Patch Generation in the Era of LLM-based Automated Program Repaircited by 4
  29. 2025EyeMulator: Improving Code Language Models by Mimicking Human Visual Attentioncited by 4
  30. 2025Collaborative Agents for Automated Program Repair in Rubycited by 1

Built on by

Not cited yet.