# Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning

arXiv: https://arxiv.org/abs/2301.01113 · HTML: https://mlsanity.com/paper/2301.01113/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 79 references that are in the map.
- First posted on arXiv 3 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022)
- [TBar: Revisiting Template-based Automated Program Repair](https://mlsanity.com/paper/1903.08409/) (2019)
- [Neural Program Repair with Execution-based Backpropagation](https://mlsanity.com/paper/2105.04123/) (2021)
- [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020)
- [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022)
- [Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs](https://mlsanity.com/paper/1811.04211/) (2018)
- [A Compare-Aggregate Model for Matching Text Sequences](https://mlsanity.com/paper/1611.01747/) (2016)
- [FixMiner: Mining Relevant Fix Patterns for Automated Program Repair](https://mlsanity.com/paper/1810.01791/) (2018)
- [AVATAR : Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations](https://mlsanity.com/paper/1812.07270/) (2018)
- [Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair](https://mlsanity.com/paper/2008.02944/) (2020)
- [A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark](https://mlsanity.com/paper/1805.03454/) (2018)
- [Identifying Patch Correctness in Test-Based Program Repair](https://mlsanity.com/paper/1706.09120/) (2017)
- [Automated Patch Assessment for Program Repair at Scale](https://mlsanity.com/paper/1909.13694/) (2019)
- [Precise Condition Synthesis for Program Repair](https://mlsanity.com/paper/1608.07754/) (2016)
- [AutoPruner: Transformer-Based Call Graph Pruning](https://mlsanity.com/paper/2209.03230/) (2022)
- [VulCurator: A Vulnerability-Fixing Commit Detector](https://mlsanity.com/paper/2209.03260/) (2022)
- [Automated Classification of Overfitting Patches with Statically Extracted Code Features](https://mlsanity.com/paper/1910.12057/) (2019)
- [On Reliability of Patch Correctness Assessment](https://mlsanity.com/paper/1805.05983/) (2018)
- [Alleviating Patch Overfitting with Automatic Test Generation: A Study of Feasibility and Effectiveness for the Nopol Repair System](https://mlsanity.com/paper/1810.10614/) (2018)
- [Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset](https://mlsanity.com/paper/1811.02429/) (2018)
- [A Critical Review of "Automatic Patch Generation Learned from Human-Written Patches": Essay on the Problem Statement and the Evaluation of Automatic Software Repair](https://mlsanity.com/paper/1408.2103/) (2014)
- [A Counterexample-guided Approach to Finding Numerical Invariants](https://mlsanity.com/paper/1903.12113/) (2019)
- [DynamiTe: Dynamic Termination and Non-termination Proofs](https://mlsanity.com/paper/2010.05747/) (2020)
- [Toward the Analysis of Graph Neural Networks](https://mlsanity.com/paper/2201.00115/) (2022)
- [Usability and Aesthetics: Better Together for Automated Repair of Web Pages](https://mlsanity.com/paper/2201.00117/) (2022)

## Built on by (most cited first, 11 of 11)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Exploring ChatGPT's Capabilities on Vulnerability Management](https://mlsanity.com/paper/2311.06530/) (2023)
- [Are LLMs Correctly Integrated into Software Systems?](https://mlsanity.com/paper/2407.05138/) (2024)
- [Automated Program Repair: Emerging trends pose and expose problems for benchmarks](https://mlsanity.com/paper/2405.05455/) (2024)
- [Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing](https://mlsanity.com/paper/2402.11892/) (2024)
- [Multi-Granularity Detector for Vulnerability Fixes](https://mlsanity.com/paper/2305.13884/) (2023)
- [Past, Present, and Future of Bug Tracking in the Generative AI Era](https://mlsanity.com/paper/2510.08005/) (2025)
- [LEGION: Harnessing Pre-trained Language Models for GitHub Topic Recommendations with Distribution-Balance Loss](https://mlsanity.com/paper/2403.05873/) (2024)
- [RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment](https://mlsanity.com/paper/2507.22580/) (2025)
- [LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software](https://mlsanity.com/paper/2604.12994/) (2026)

## Abstract (arXiv)

Automated program repair (APR) faces the challenge of test overfitting, where generated patches pass validation tests but fail to generalize. Existing methods for patch assessment involve generating new tests or manual inspection, which can be time-consuming or biased. In this paper, we propose a novel technique, INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR leverages program invariants to reason about program semantics while also capturing program syntax through language semantics learned from a large code corpus using a pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that an APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains erroneous behaviors from the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is threefold. First, INVALIDATOR leverages both semantic and syntactic reasoning to enhance its discriminative capability. Second, INVALIDATOR does not require new test cases to be generated, but instead only relies on the current test suite and uses invariant inference to generalize program behaviors. Third, INVALIDATOR is fully automated. Experimental results demonstrate that INVALIDATOR outperforms existing methods in terms of Accuracy and F-measure, correctly identifying 79% of overfitting patches and detecting 23% more overfitting patches than the best baseline.