# Identifying Patch Correctness in Test-Based Program Repair

arXiv: https://arxiv.org/abs/1706.09120 · HTML: https://mlsanity.com/paper/1706.09120/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2017.

## Built on by (most cited first, 17 of 17)

- [Evaluating Agent-based Program Repair at Google](https://mlsanity.com/paper/2501.07531/) (2025)
- [Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study](https://mlsanity.com/paper/2506.08311/) (2025)
- [Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions](https://mlsanity.com/paper/2304.03816/) (2023)
- [Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning](https://mlsanity.com/paper/2301.01113/) (2023)
- [RunBugRun -- An Executable Dataset for Automated Program Repair](https://mlsanity.com/paper/2304.01102/) (2023)
- [Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness](https://mlsanity.com/paper/2208.04125/) (2022)
- [Predicting Patch Correctness Based on the Similarity of Failing Test Cases](https://mlsanity.com/paper/2107.13296/) (2021)
- [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026)
- [PatchBench: Evaluating AI Agents for Vulnerability Patching](https://mlsanity.com/paper/2609.04075/) (2026)
- [RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment](https://mlsanity.com/paper/2507.22580/) (2025)
- [Coupling Planning with Episodic Memory in LLM Agents for Software Issue Resolution](https://mlsanity.com/paper/2608.06811/) (2026)
- [BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure](https://mlsanity.com/paper/2609.11028/) (2026)
- [Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents](https://mlsanity.com/paper/2610.01023/) (2026)
- [Exploring Plausible Patches Using Source Code Embeddings in JavaScript](https://mlsanity.com/paper/2103.16846/) (2021)
- [Understanding Agent-Based Patching of Compiler Missed Optimizations](https://mlsanity.com/paper/2607.02370/) (2026)
- [From Approval to Execution: Reconstruction-Aware Repair Analysis for LLM-Agent Software](https://mlsanity.com/paper/2609.26529/) (2026)
- [P2Patch: Exploit-Guided Vulnerability Repair](https://mlsanity.com/paper/2610.20245/) (2026)

## Abstract (arXiv)

Test-based automatic program repair has attracted a lot of attention in recent years. However, the test suites in practice are often too weak to guarantee correctness and existing approaches often generate a large number of incorrect patches. To reduce the number of incorrect patches generated, we propose a novel approach that heuristically determines the correctness of the generated patches. The core idea is to exploit the behavior similarity of test case executions. The passing tests on original and patched programs are likely to behave similarly while the failing tests on original and patched programs are likely to behave differently. Also, if two tests exhibit similar runtime behavior, the two tests are likely to have the same test results. Based on these observations, we generate new test inputs to enhance the test suites and use their behavior similarity to determine patch correctness. Our approach is evaluated on a dataset consisting of 139 patches generated from existing program repair systems including jGenProg, Nopol, jKali, ACS and HDRepair. Our approach successfully prevented 56.3% of the incorrect patches to be generated, without blocking any correct patches.