# A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis

arXiv: https://arxiv.org/abs/2512.19481 · HTML: https://mlsanity.com/paper/2512.19481/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 22 Dec 2025.

## Builds on (most cited first, 8 of 8)

- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [Towards Automating Code Review Activities](https://mlsanity.com/paper/2101.02518/) (2021)
- [Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead](https://mlsanity.com/paper/2404.02525/) (2024)
- [Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?](https://mlsanity.com/paper/2411.02093/) (2024)
- [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023)
- [Exploring the Capabilities of LLMs for Code Change Related Tasks](https://mlsanity.com/paper/2407.02824/) (2024)
- [Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Models](https://mlsanity.com/paper/2502.06039/) (2025)

## Abstract (arXiv)

Understanding source code changes and their impact on other code entities is a crucial skill in software development. However, the analysis of code changes and their impact is often performed manually and therefore is time-consuming. Recent advancements in AI, and in particular large language models (LLMs) show promises to help developers in various code analysis tasks. However, the extent to which this potential can be utilized for understanding code changes and their impact is underexplored. To address this gap, we study the capabilities of GPT-5 and GPT-5-mini to predict the code entities impacted by given source code changes. We construct a dataset containing information about seed-changes, change pairs, and change types for each commit. Existing datasets lack crucial information about seed changes and impacted code entities. Our experiments evaluate the LLMs in two configurations: (1) seed-change information and the parent commit tree and (2) seed-change information, the parent commit tree, and the diff hunk of each seed change. We found that both LLMs perform poorly in the two experiments, whereas GPT-5 outperforms GPT-5-mini. Furthermore, the provision of the diff hunks helps both models to slightly improve their performance.