# Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair

arXiv: https://arxiv.org/abs/2008.02944 · HTML: https://mlsanity.com/paper/2008.02944/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 22 of 22)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Baldur: Whole-Proof Generation and Repair with Large Language Models](https://mlsanity.com/paper/2303.04910/) (2023)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures](https://mlsanity.com/paper/2509.14252/) (2025)
- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models](https://mlsanity.com/paper/2404.18353/) (2024)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [DeepCVA: Automated Commit-level Vulnerability Assessment with Deep Multi-task Learning](https://mlsanity.com/paper/2108.08041/) (2021)
- [Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning](https://mlsanity.com/paper/2301.01113/) (2023)
- [Generative AI for Requirements Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2409.06741/) (2024)
- [Extending the Frontier of ChatGPT: Code Generation and Debugging](https://mlsanity.com/paper/2307.08260/) (2023)
- [Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness](https://mlsanity.com/paper/2208.04125/) (2022)
- [Just-in-Time Detection of Silent Security Patches](https://mlsanity.com/paper/2312.01241/) (2023)
- [Automated Program Repair: Emerging trends pose and expose problems for benchmarks](https://mlsanity.com/paper/2405.05455/) (2024)
- [On the Use of Fine-grained Vulnerable Code Statements for Software Vulnerability Assessment Models](https://mlsanity.com/paper/2203.08417/) (2022)
- [Predicting Patch Correctness Based on the Similarity of Failing Test Cases](https://mlsanity.com/paper/2107.13296/) (2021)
- [CCRep: Learning Code Change Representations via Pre-Trained Code Model and Query Back](https://mlsanity.com/paper/2302.03924/) (2023)
- [An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code](https://mlsanity.com/paper/2307.02443/) (2023)
- [RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment](https://mlsanity.com/paper/2507.22580/) (2025)
- [Exploring Plausible Patches Using Source Code Embeddings in JavaScript](https://mlsanity.com/paper/2103.16846/) (2021)
- [Transducer Tuning: Efficient Model Adaptation for Software Tasks Using Code Property Graphs](https://mlsanity.com/paper/2412.13467/) (2024)

## Abstract (arXiv)

A large body of the literature of automated program repair develops approaches where patches are generated to be validated against an oracle (e.g., a test suite). Because such an oracle can be imperfect, the generated patches, although validated by the oracle, may actually be incorrect. While the state of the art explore research directions that require dynamic information or rely on manually-crafted heuristics, we study the benefit of learning code representations to learn deep features that may encode the properties of patch correctness. Our work mainly investigates different representation learning approaches for code changes to derive embeddings that are amenable to similarity computations. We report on findings based on embeddings produced by pre-trained and re-trained neural networks. Experimental results demonstrate the potential of embeddings to empower learning algorithms in reasoning about patch correctness: a machine learning predictor with BERT transformer-based embeddings associated with logistic regression yielded an AUC value of about 0.8 in predicting patch correctness on a deduplicated dataset of 1000 labeled patches. Our study shows that learned representations can lead to reasonable performance when comparing against the state-of-the-art, PATCH-SIM, which relies on dynamic information. These representations may further be complementary to features that were carefully (manually) engineered in the literature.