# SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

arXiv: https://arxiv.org/abs/2607.13111 · HTML: https://mlsanity.com/paper/2607.13111/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 14 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Test-Driven Development for Code Generation](https://mlsanity.com/paper/2402.13521/) (2024)
- [Untangling Fine-Grained Code Changes](https://mlsanity.com/paper/1502.06757/) (2015)
- [ChangeGuard: Validating Code Changes via Pairwise Learning-Guided Execution](https://mlsanity.com/paper/2410.16092/) (2024)
- [RefDiff: Detecting Refactorings in Version Histories](https://mlsanity.com/paper/1704.01544/) (2017)
- [On the Relationship between Refactoring Actions and Bugs: A Differentiated Replication](https://mlsanity.com/paper/2009.11685/) (2020)
- [Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approach](https://mlsanity.com/paper/2404.06572/) (2024)
- [A Differential Fuzzing-Based Evaluation of Functional Equivalence in LLM-Generated Code Refactorings](https://mlsanity.com/paper/2602.15761/) (2026)
- [How well LLM-based test generation techniques perform with newer LLM versions?](https://mlsanity.com/paper/2601.09695/) (2026)

## Built on by (most cited first, 1 of 1)

- [Impact Is Not Invalidation: Ask About the Claim, Not the Diff](https://mlsanity.com/paper/2609.25130/) (2026)

## Abstract (arXiv)

Distinguishing semantic-preserving commits from changing ones remains an open challenge in software repository mining. While existing approaches detect refactoring commits accurately, they cannot ensure that a commit is purely semantic-preserving, without any interleaving behaviour-changing modification. This limitation can impact several tasks, such as debugging, fault localisation, bug dataset construction, rollback analysis, and bug fixes backporting. To fill this gap, we propose SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis; comparison of similar test execution on pre- and post-commit versions. As code impacted by the refactoring is often hard to test and different accross both versions, we propose generating additional calling methods to that code, which serve as testing target. Given a commit, SemaDiff analyses the diff to identify modified code and extracts unchanged dependent code that calls it. It then generates an additional dependent class using a large language model to exercise the changed code in both versions, and automatically generates tests for the dependent code. This way, we obtain the same tests for the different code versions, enabling the behavioural-difference detection. The commit is classified as semantic-preserving only if all generated tests produce identical outcomes across the two versions. To evaluate SemaDiff, we construct and annotate manually a dataset of 183 commits, gathered from well-known open-source Java projects. The obtained results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.