# Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing arXiv: https://arxiv.org/abs/2402.11892 ยท HTML: https://mlsanity.com/paper/2402.11892/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 43 of its 106 references that are in the map. - First posted on arXiv 19 Feb 2024. ## Builds on (most cited first, 30 of 43) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017) - [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018) - [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020) - [An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation](https://mlsanity.com/paper/1812.08693/) (2018) - [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021) - [Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues](https://mlsanity.com/paper/2307.12596/) (2023) - [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022) - [TBar: Revisiting Template-based Automated Program Repair](https://mlsanity.com/paper/1903.08409/) (2019) - [A Syntax-Guided Edit Decoder for Neural Program Repair](https://mlsanity.com/paper/2106.08253/) (2021) - [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021) - [Neural Program Repair with Execution-based Backpropagation](https://mlsanity.com/paper/2105.04123/) (2021) - [RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair](https://mlsanity.com/paper/2312.15698/) (2023) - [A Systematic Review of Robustness in Deep Learning for Computer Vision: Mind the gap?](https://mlsanity.com/paper/2112.00639/) (2021) - [On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations](https://mlsanity.com/paper/2008.01566/) (2020) - [CODIT: Code Editing with Tree-Based Neural Models](https://mlsanity.com/paper/1810.00314/) (2018) - [BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies](https://mlsanity.com/paper/2401.15481/) (2024) - [On the "Naturalness" of Buggy Code](https://mlsanity.com/paper/1506.01159/) (2015) - [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022) - [Trust Enhancement Issues in Program Repair](https://mlsanity.com/paper/2108.13064/) (2021) - [SPI: Automated Identification of Security Patches via Commits](https://mlsanity.com/paper/2105.14565/) (2021) - [Patching as Translation: the Data and the Metaphor](https://mlsanity.com/paper/2008.10707/) (2020) - [How do humans perceive adversarial text? A reality check on the validity and naturalness of word-based adversarial attacks](https://mlsanity.com/paper/2305.15587/) (2023) - [On the Efficiency of Test Suite based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programs](https://mlsanity.com/paper/2008.00914/) (2020) - [Challenging Machine Learning-based Clone Detectors via Semantic-preserving Code Transformations](https://mlsanity.com/paper/2111.10793/) (2021) ## Built on by (most cited first, 5 of 5) - [Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference](https://mlsanity.com/paper/2503.04779/) (2025) - [Memory-Efficient Large Language Models for Program Repair with Semantic-Guided Patch Generation](https://mlsanity.com/paper/2410.16655/) (2024) - [DynaFix: Iterative Automated Program Repair Driven by Execution-Level Dynamic Information](https://mlsanity.com/paper/2512.24635/) (2025) - [Auto-SPT: Automating Semantic Preserving Transformations for Code](https://mlsanity.com/paper/2512.06042/) (2025) - [How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?](https://mlsanity.com/paper/2512.15468/) (2025) ## Abstract (arXiv) In this paper, we propose shifting the focus of robustness evaluation for Neural Program Repair (NPR) techniques toward naturally-occurring data transformations. To accomplish this, we first examine the naturalness of semantic-preserving transformations through a two-stage human study. This study includes (1) interviews with senior software developers to establish concrete criteria for evaluating the naturalness of these transformations, and (2) a survey involving 10 developers to assess the naturalness of 1,178 transformations, i.e., pairs of original and transformed programs, applied to 225 real-world bugs. Our findings show that only 60% of these transformations are deemed natural, while 20% are considered unnatural, with strong agreement among annotators. Moreover, the unnaturalness of these transformations significantly impacts both their applicability to benchmarks and the conclusions drawn from robustness testing. Next, we conduct natural robustness testing on NPR techniques to assess their true effectiveness against real-world data variations. Our experimental results reveal a substantial number of prediction changes in NPR techniques, leading to significant reductions in both plausible and correct patch rates when comparing performance on the original and transformed datasets. Additionally, we observe notable differences in performance improvements between NPR techniques, suggesting potential biases on NPR evaluation introduced by limited datasets. Finally, we propose an LLM-based metric to automate the assessment of transformation naturalness, ensuring the scalability of natural robustness testing.