# CCBERT: Self-Supervised Code Change Representation Learning

arXiv: https://arxiv.org/abs/2309.15474 · HTML: https://mlsanity.com/paper/2309.15474/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2023.

## Built on by (most cited first, 6 of 6)

- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures](https://mlsanity.com/paper/2509.14252/) (2025)
- [Just-in-Time Detection of Silent Security Patches](https://mlsanity.com/paper/2312.01241/) (2023)
- [LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation](https://mlsanity.com/paper/2501.08558/) (2025)
- [Just-In-Time Software Defect Prediction via Bi-modal Change Representation Learning](https://mlsanity.com/paper/2410.12107/) (2024)

## Abstract (arXiv)

Numerous code changes are made by developers in their daily work, and a superior representation of code changes is desired for effective code change analysis. Recently, Hoang et al. proposed CC2Vec, a neural network-based approach that learns a distributed representation of code changes to capture the semantic intent of the changes. Despite demonstrated effectiveness in multiple tasks, CC2Vec has several limitations: 1) it considers only coarse-grained information about code changes, and 2) it relies on log messages rather than the self-contained content of the code changes. In this work, we propose CCBERT (underlineCode underlineChange underlineBERT), a new Transformer-based pre-trained model that learns a generic representation of code changes based on a large-scale dataset containing massive unlabeled code changes. CCBERT is pre-trained on four proposed self-supervised objectives that are specialized for learning code change representations based on the contents of code changes. CCBERT perceives fine-grained code changes at the token level by learning from the old and new versions of the content, along with the edit actions. Our experiments demonstrate that CCBERT significantly outperforms CC2Vec or the state-of-the-art approaches of the downstream tasks by 7.7%--14.0% in terms of different metrics and tasks. CCBERT consistently outperforms large pre-trained code models, such as CodeBERT, while requiring 6--10× less training time, 5--30× less inference time, and 7.9× less GPU memory.