# ReDef: Do Code Language Models Truly Understand Code Changes for Just-in-Time Software Defect Prediction?

arXiv: https://arxiv.org/abs/2509.09192 · HTML: https://mlsanity.com/paper/2509.09192/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 74 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020)
- [Automatic Code Summarization via ChatGPT: How Far Are We?](https://mlsanity.com/paper/2305.12865/) (2023)
- [An Empirical Comparison of Pre-Trained Models of Source Code](https://mlsanity.com/paper/2302.04026/) (2023)
- [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023)
- [Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models](https://mlsanity.com/paper/2310.11113/) (2023)
- [Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models](https://mlsanity.com/paper/2401.01060/) (2024)
- [Evaluating SZZ Implementations: An Empirical Study on the Linux Kernel](https://mlsanity.com/paper/2308.05060/) (2023)
- [Problems with SZZ and Features: An empirical study of the state of practice of defect prediction data collection](https://mlsanity.com/paper/1911.08938/) (2019)
- [Tram: A Token-level Retrieval-augmented Mechanism for Source Code Summarization](https://mlsanity.com/paper/2305.11074/) (2023)
- [The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification](https://mlsanity.com/paper/2305.04940/) (2023)
- [Benchmarking Large Language Models for Multi-Language Software Vulnerability Detection](https://mlsanity.com/paper/2503.01449/) (2025)
- [Line-level Semantic Structure Learning for Code Vulnerability Detection](https://mlsanity.com/paper/2407.18877/) (2024)
- [A study on the impact of pre-trained model on Just-In-Time defect prediction](https://mlsanity.com/paper/2309.02317/) (2023)
- [Empirical Study on Transformer-based Techniques for Software Engineering](https://mlsanity.com/paper/2310.00399/) (2023)
- [Refactoring neq Bug-Inducing: Improving Defect Prediction with Code Change Tactics Analysis](https://mlsanity.com/paper/2507.19714/) (2025)
- [Just-In-Time Software Defect Prediction via Bi-modal Change Representation Learning](https://mlsanity.com/paper/2410.12107/) (2024)

## Abstract (arXiv)

Just-in-Time software defect prediction (JIT-SDP) plays a critical role in prioritizing risky code changes during code review and continuous integration. However, existing datasets often suffer from noisy labels and low precision in identifying bug-inducing commits. To address this, we present ReDef (Revert-based Defect dataset), a high-confidence benchmark of function-level modifications curated from 22 large-scale C/C++ projects. Defective cases are anchored by revert commits, while clean cases are validated through post-hoc history checks. Ambiguous instances are conservatively filtered out via a GPT-assisted triage process involving multiple votes and audits. This pipeline yields 3,164 defective and 10,268 clean modifications, offering substantially more reliable labels than prior resources. Beyond dataset construction, we provide a systematic evaluation of how Code Language Models (CLMs)-specifically CodeBERT, CodeT5+, UniXcoder, and Qwen2.5-reason about code modifications. We first investigate which input encodings most effectively expose change information under five different strategies. We then design four counterfactual perturbation strategies (e.g., swapping added/deleted blocks, inverting diff polarity) to serve as diagnostic probes. We posit that if models genuinely capture change semantics, such distortions should lead to a clear decline in predictive performance. Our results show that compact diff-style encodings consistently outperform whole-function formats across all CLMs, supported by rigorous statistical confirmation. However, under counterfactual tests, performance remains effectively stable, revealing that what appears to be robustness in fact reflects a reliance on superficial cues rather than true semantic understanding.