# Just-In-Time Software Defect Prediction via Bi-modal Change Representation Learning

arXiv: https://arxiv.org/abs/2410.12107 · HTML: https://mlsanity.com/paper/2410.12107/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 63 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [CoditT5: Pretraining for Source Code and Natural Language Editing](https://mlsanity.com/paper/2208.05446/) (2022)
- [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020)
- [Learning to Represent Edits](https://mlsanity.com/paper/1810.13337/) (2018)
- [Learning Structural Edits via Incremental Tree Transformations](https://mlsanity.com/paper/2101.12087/) (2021)
- [CoreGen: Contextualized Code Representation Learning for Commit Message Generation](https://mlsanity.com/paper/2007.06934/) (2020)
- [JITLine: A Simpler, Better, Faster, Finer-grained Just-In-Time Defect Prediction](https://mlsanity.com/paper/2103.07068/) (2021)
- [Copy that! Editing Sequences by Copying Spans](https://mlsanity.com/paper/2006.04771/) (2020)
- [Multi-timescale Representation Learning in LSTM Language Models](https://mlsanity.com/paper/2009.12727/) (2020)
- [Commit2Vec: Learning Distributed Representations of Code Changes](https://mlsanity.com/paper/1911.07605/) (2019)
- [A Fine-grained Data Set and Analysis of Tangling in Bug Fixing Commits](https://mlsanity.com/paper/2011.06244/) (2020)
- [Code-centric Learning-based Just-In-Time Vulnerability Detection](https://mlsanity.com/paper/2304.08396/) (2023)
- [Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study](https://mlsanity.com/paper/2402.06247/) (2024)
- [On the Use of Deep Learning in Software Defect Prediction](https://mlsanity.com/paper/2210.02236/) (2022)
- [CCBERT: Self-Supervised Code Change Representation Learning](https://mlsanity.com/paper/2309.15474/) (2023)
- [Unsupervised Learning of General-Purpose Embeddings for Code Changes](https://mlsanity.com/paper/2106.02087/) (2021)
- [Zero-Shot Program Representation Learning](https://mlsanity.com/paper/2204.08360/) (2022)
- [Context-Encoded Code Change Representation for Automated Commit Message Generation](https://mlsanity.com/paper/2306.14418/) (2023)
- [Bridging Expert Knowledge with Deep Learning Techniques for Just-In-Time Defect Prediction](https://mlsanity.com/paper/2403.11079/) (2024)
- [Human-in-the-loop online just-in-time software defect prediction](https://mlsanity.com/paper/2308.13707/) (2023)

## Built on by (most cited first, 1 of 1)

- [ReDef: Do Code Language Models Truly Understand Code Changes for Just-in-Time Software Defect Prediction?](https://mlsanity.com/paper/2509.09192/) (2025)

## Abstract (arXiv)

For predicting software defects at an early stage, researchers have proposed just-in-time defect prediction (JIT-DP) to identify potential defects in code commits. The prevailing approaches train models to represent code changes in history commits and utilize the learned representations to predict the presence of defects in the latest commit. However, existing models merely learn editions in source code, without considering the natural language intentions behind the changes. This limitation hinders their ability to capture deeper semantics. To address this, we introduce a novel bi-modal change pre-training model called BiCC-BERT. BiCC-BERT is pre-trained on a code change corpus to learn bi-modal semantic representations. To incorporate commit messages from the corpus, we design a novel pre-training objective called Replaced Message Identification (RMI), which learns the semantic association between commit messages and code changes. Subsequently, we integrate BiCC-BERT into JIT-DP and propose a new defect prediction approach -- JIT-BiCC. By leveraging the bi-modal representations from BiCC-BERT, JIT-BiCC captures more profound change semantics. We train JIT-BiCC using 27,391 code changes and compare its performance with 8 state-of-the-art JIT-DP approaches. The results demonstrate that JIT-BiCC outperforms all baselines, achieving a 10.8% improvement in F1-score. This highlights its effectiveness in learning the bi-modal semantics for JIT-DP.