# CCRep: Learning Code Change Representations via Pre-Trained Code Model and Query Back

arXiv: https://arxiv.org/abs/2302.03924 · HTML: https://mlsanity.com/paper/2302.03924/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 68 references that are in the map.
- First posted on arXiv 8 Feb 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [Multi-task Learning based Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2012.14631/) (2020)
- [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](https://mlsanity.com/paper/2003.07914/) (2020)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020)
- [How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset](https://mlsanity.com/paper/1905.13334/) (2019)
- [Learning to Represent Edits](https://mlsanity.com/paper/1810.13337/) (2018)
- [Learning to Update Natural Language Comments Based on Code Changes](https://mlsanity.com/paper/2004.12169/) (2020)
- [ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking](https://mlsanity.com/paper/1912.02972/) (2019)
- [Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair](https://mlsanity.com/paper/2008.02944/) (2020)
- [Deep Just-In-Time Inconsistency Detection Between Comments and Source Code](https://mlsanity.com/paper/2010.01625/) (2020)
- [Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models](https://mlsanity.com/paper/2102.04411/) (2021)
- [On the Evaluation of Commit Message Generation Models: An Experimental Study](https://mlsanity.com/paper/2107.05373/) (2021)
- [SPI: Automated Identification of Security Patches via Commits](https://mlsanity.com/paper/2105.14565/) (2021)
- [Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2,141 Bugs and 23,551 Repair Attempts](https://mlsanity.com/paper/1905.11973/) (2019)
- [DeepCVA: Automated Commit-level Vulnerability Assessment with Deep Multi-task Learning](https://mlsanity.com/paper/2108.08041/) (2021)
- [Untangling Fine-Grained Code Changes](https://mlsanity.com/paper/1502.06757/) (2015)

## Built on by (most cited first, 3 of 3)

- [Just-in-Time Detection of Silent Security Patches](https://mlsanity.com/paper/2312.01241/) (2023)
- [The Rise and Fall(?) of Software Engineering](https://mlsanity.com/paper/2406.10141/) (2024)
- [AI-Powered Commit Explorer (APCE)](https://mlsanity.com/paper/2507.16063/) (2025)

## Abstract (arXiv)

Representing code changes as numeric feature vectors, i.e., code change representations, is usually an essential step to automate many software engineering tasks related to code changes, e.g., commit message generation and just-in-time defect prediction. Intuitively, the quality of code change representations is crucial for the effectiveness of automated approaches. Prior work on code changes usually designs and evaluates code change representation approaches for a specific task, and little work has investigated code change encoders that can be used and jointly trained on various tasks. To fill this gap, this work proposes a novel Code Change Representation learning approach named CCRep, which can learn to encode code changes as feature vectors for diverse downstream tasks. Specifically, CCRep regards a code change as the combination of its before-change and after-change code, leverages a pre-trained code model to obtain high-quality contextual embeddings of code, and uses a novel mechanism named query back to extract and encode the changed code fragments and make them explicitly interact with the whole code change. To evaluate CCRep and demonstrate its applicability to diverse code-change-related tasks, we apply it to three tasks: commit message generation, patch correctness assessment, and just-in-time defect prediction. Experimental results show that CCRep outperforms the state-of-the-art techniques on each task.