# Automated Patch Assessment for Program Repair at Scale

arXiv: https://arxiv.org/abs/1909.13694 · HTML: https://mlsanity.com/paper/1909.13694/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 14 of 14)

- [Baldur: Whole-Proof Generation and Repair with Large Language Models](https://mlsanity.com/paper/2303.04910/) (2023)
- [An Empirical Study on Failures in Automated Issue Solving](https://mlsanity.com/paper/2509.13941/) (2025)
- [Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning](https://mlsanity.com/paper/2301.01113/) (2023)
- [RunBugRun -- An Executable Dataset for Automated Program Repair](https://mlsanity.com/paper/2304.01102/) (2023)
- [Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness](https://mlsanity.com/paper/2208.04125/) (2022)
- [Automated Program Repair: Emerging trends pose and expose problems for benchmarks](https://mlsanity.com/paper/2405.05455/) (2024)
- [Predicting Patch Correctness Based on the Similarity of Failing Test Cases](https://mlsanity.com/paper/2107.13296/) (2021)
- [Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model](https://mlsanity.com/paper/2608.13867/) (2026)
- [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026)
- [Risks of ignoring uncertainty propagation in AI-augmented security pipelines](https://mlsanity.com/paper/2407.14540/) (2024)
- [RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment](https://mlsanity.com/paper/2507.22580/) (2025)
- [VeriSkill: A Self-Evolution Framework for Program Verification Skills](https://mlsanity.com/paper/2607.27733/) (2026)
- [CAPRI: Contract-Aware Proof Repair for Isabelle](https://mlsanity.com/paper/2608.13459/) (2026)
- [P2Patch: Exploit-Guided Vulnerability Repair](https://mlsanity.com/paper/2610.20245/) (2026)

## Abstract (arXiv)

In this paper, we do automatic correctness assessment for patches generated by program repair systems. We consider the human-written patch as ground truth oracle and randomly generate tests based on it, a technique proposed by Shamshiri et al., called Random testing with Ground Truth (RGT) in this paper. We build a curated dataset of 638 patches for Defects4J generated by 14 state-of-the-art repair systems, we evaluate automated patch assessment on this dataset. The results of this study are novel and significant: First, we improve the state of the art performance of automatic patch assessment with RGT by 190% by improving the oracle; Second, we show that RGT is reliable enough to help scientists to do overfitting analysis when they evaluate program repair systems; Third, we improve the external validity of the program repair knowledge with the largest study ever.