# Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness

arXiv: https://arxiv.org/abs/2208.04125 · HTML: https://mlsanity.com/paper/2208.04125/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 69 references that are in the map.
- First posted on arXiv 8 Aug 2022.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation](https://mlsanity.com/paper/1812.08693/) (2018)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [Neural Program Repair with Execution-based Backpropagation](https://mlsanity.com/paper/2105.04123/) (2021)
- [Applying CodeBERT for Automated Program Repair of Java Simple Bugs](https://mlsanity.com/paper/2103.11626/) (2021)
- [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020)
- [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022)
- [CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2104.02443/) (2021)
- [AVATAR : Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations](https://mlsanity.com/paper/1812.07270/) (2018)
- [Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair](https://mlsanity.com/paper/2008.02944/) (2020)
- [LSTM-based Deep Learning Models for Non-factoid Answer Selection](https://mlsanity.com/paper/1511.04108/) (2015)
- [Identifying Patch Correctness in Test-Based Program Repair](https://mlsanity.com/paper/1706.09120/) (2017)
- [On the Efficiency of Test Suite based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programs](https://mlsanity.com/paper/2008.00914/) (2020)
- [Automated Patch Assessment for Program Repair at Scale](https://mlsanity.com/paper/1909.13694/) (2019)
- [Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies](https://mlsanity.com/paper/1901.06024/) (2019)
- [Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2,141 Bugs and 23,551 Repair Attempts](https://mlsanity.com/paper/1905.11973/) (2019)
- [iFixR: Bug Report driven Program Repair](https://mlsanity.com/paper/1907.05620/) (2019)
- [Automated Classification of Overfitting Patches with Statically Extracted Code Features](https://mlsanity.com/paper/1910.12057/) (2019)
- [An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems](https://mlsanity.com/paper/1602.05643/) (2016)
- [Question Answering through Transfer Learning from Large Fine-grained Supervision Data](https://mlsanity.com/paper/1702.02171/) (2017)
- [The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches](https://mlsanity.com/paper/2203.08912/) (2022)
- [Alleviating Patch Overfitting with Automatic Test Generation: A Study of Feasibility and Effectiveness for the Nopol Repair System](https://mlsanity.com/paper/1810.10614/) (2018)
- [Predicting Patch Correctness Based on the Similarity of Failing Test Cases](https://mlsanity.com/paper/2107.13296/) (2021)
- [How Different Is It Between Machine-Generated and Developer-Provided Patches? An Empirical Study on The Correct Patches Generated by Automated Program Repair Techniques](https://mlsanity.com/paper/1906.03447/) (2019)

## Built on by (most cited first, 7 of 7)

- [CodeAgent: Autonomous Communicative Agents for Code Review](https://mlsanity.com/paper/2402.02172/) (2024)
- [Exploring ChatGPT's Capabilities on Vulnerability Management](https://mlsanity.com/paper/2311.06530/) (2023)
- [Just-in-Time Detection of Silent Security Patches](https://mlsanity.com/paper/2312.01241/) (2023)
- [Automated Program Repair: Emerging trends pose and expose problems for benchmarks](https://mlsanity.com/paper/2405.05455/) (2024)
- [RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment](https://mlsanity.com/paper/2507.22580/) (2025)
- [Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review](https://mlsanity.com/paper/2605.17548/) (2026)
- [LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software](https://mlsanity.com/paper/2604.12994/) (2026)

## Abstract (arXiv)

In this work, we propose a novel perspective to the problem of patch correctness assessment: a correct patch implements changes that "answer" to a problem posed by buggy behaviour. Concretely, we turn the patch correctness assessment into a Question Answering problem. To tackle this problem, our intuition is that natural language processing can provide the necessary representations and models for assessing the semantic correlation between a bug (question) and a patch (answer). Specifically, we consider as inputs the bug reports as well as the natural language description of the generated patches. Our approach, Quatrain, first considers state of the art commit message generation models to produce the relevant inputs associated to each generated patch. Then we leverage a neural network architecture to learn the semantic correlation between bug reports and commit messages. Experiments on a large dataset of 9135 patches generated for three bug datasets (Defects4j, Bugs.jar and Bears) show that Quatrain can achieve an AUC of 0.886 on predicting patch correctness, and recalling 93% correct patches while filtering out 62% incorrect patches. Our experimental results further demonstrate the influence of inputs quality on prediction performance. We further perform experiments to highlight that the model indeed learns the relationship between bug reports and code change descriptions for the prediction. Finally, we compare against prior work and discuss the benefits of our approach.