# Unsupervised Learning of General-Purpose Embeddings for Code Changes

arXiv: https://arxiv.org/abs/2106.02087 · HTML: https://mlsanity.com/paper/2106.02087/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [On Learning Meaningful Code Changes via Neural Machine Translation](https://mlsanity.com/paper/1901.09102/) (2019)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016)
- [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020)
- [Learning to Represent Edits](https://mlsanity.com/paper/1810.13337/) (2018)
- [Learning and Evaluating Contextual Embedding of Source Code](https://mlsanity.com/paper/2001.00059/) (2019)
- [A Literature Study of Embeddings on Source Code](https://mlsanity.com/paper/1904.03061/) (2019)
- [Commit2Vec: Learning Distributed Representations of Code Changes](https://mlsanity.com/paper/1911.07605/) (2019)
- [PatchNet: Hierarchical Deep Learning-Based Stable Patch Identification for the Linux Kernel](https://mlsanity.com/paper/1911.03576/) (2019)
- [Generating Commit Messages from Git Diffs](https://mlsanity.com/paper/1911.11690/) (2019)

## Built on by (most cited first, 3 of 3)

- [CommitBench: A Benchmark for Commit Message Generation](https://mlsanity.com/paper/2403.05188/) (2024)
- [Just-In-Time Software Defect Prediction via Bi-modal Change Representation Learning](https://mlsanity.com/paper/2410.12107/) (2024)
- [UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models](https://mlsanity.com/paper/2512.17385/) (2025)

## Abstract (arXiv)

Applying machine learning to tasks that operate with code changes requires their numerical representation. In this work, we propose an approach for obtaining such representations during pre-training and evaluate them on two different downstream tasks - applying changes to code and commit message generation. During pre-training, the model learns to apply the given code change in a correct way. This task requires only code changes themselves, which makes it unsupervised. In the task of applying code changes, our model outperforms baseline models by 5.9 percentage points in accuracy. As for the commit message generation, our model demonstrated the same results as supervised models trained for this specific task, which indicates that it can encode code changes well and can be improved in the future by pre-training on a larger dataset of easily gathered code changes.