# Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations

arXiv: https://arxiv.org/abs/2004.07313 · HTML: https://mlsanity.com/paper/2004.07313/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2020.

## Built on by (most cited first, 9 of 9)

- [Generating Adversarial Computer Programs using Optimized Obfuscations](https://mlsanity.com/paper/2103.11882/) (2021)
- [Understanding Neural Code Intelligence Through Program Simplification](https://mlsanity.com/paper/2106.03353/) (2021)
- [Memorization and Generalization in Neural Code Intelligence Models](https://mlsanity.com/paper/2106.08704/) (2021)
- [Syntax-Guided Program Reduction for Understanding Neural Code Intelligence Models](https://mlsanity.com/paper/2205.14374/) (2022)
- [CLAWSAT: Towards Both Robust and Accurate Code Models](https://mlsanity.com/paper/2211.11711/) (2022)
- [Study of Distractors in Neural Models of Code](https://mlsanity.com/paper/2303.01739/) (2023)
- [Coarse-Tuning Models of Code with Reinforcement Learning Feedback](https://mlsanity.com/paper/2305.18341/) (2023)
- [Assessing Code Understanding in LLMs](https://mlsanity.com/paper/2504.00065/) (2025)
- [Extracting Label-specific Key Input Features for Neural Code Intelligence Models](https://mlsanity.com/paper/2202.06474/) (2022)

## Abstract (arXiv)

The abundance of publicly available source code repositories, in conjunction with the advances in neural networks, has enabled data-driven approaches to program analysis. These approaches, called neural program analyzers, use neural networks to extract patterns in the programs for tasks ranging from development productivity to program reasoning. Despite the growing popularity of neural program analyzers, the extent to which their results are generalizable is unknown. In this paper, we perform a large-scale evaluation of the generalizability of two popular neural program analyzers using seven semantically-equivalent transformations of programs. Our results caution that in many cases the neural program analyzers fail to generalize well, sometimes to programs with negligible textual differences. The results provide the initial stepping stones for quantifying robustness in neural program analyzers.