# Understanding Neural Code Intelligence Through Program Simplification arXiv: https://arxiv.org/abs/2106.03353 ยท HTML: https://mlsanity.com/paper/2106.03353/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 45 references that are in the map. - First posted on arXiv 7 Jun 2021. ## Builds on (most cited first, 28 of 28) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014) - [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016) - [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017) - [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016) - [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013) - [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013) - [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019) - [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019) - [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019) - [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018) - [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017) - [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017) - [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018) - [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017) - [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016) - [Semantic Robustness of Models of Source Code](https://mlsanity.com/paper/2002.03043/) (2020) - [On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations](https://mlsanity.com/paper/2008.01566/) (2020) - [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016) - [Neural Program Repair by Jointly Learning to Localize and Repair](https://mlsanity.com/paper/1904.01720/) (2019) - [Adversarial Robustness for Code](https://mlsanity.com/paper/2002.04694/) (2020) - [How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset](https://mlsanity.com/paper/1905.13334/) (2019) - [On the "Naturalness" of Buggy Code](https://mlsanity.com/paper/1506.01159/) (2015) - [Learning and Evaluating Contextual Embedding of Source Code](https://mlsanity.com/paper/2001.00059/) (2019) - [Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications](https://mlsanity.com/paper/2003.07631/) (2020) - [Probing Model Signal-Awareness via Prediction-Preserving Input Minimization](https://mlsanity.com/paper/2011.14934/) (2020) - [Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations](https://mlsanity.com/paper/2004.07313/) (2020) - [Towards Demystifying Dimensions of Source Code Embeddings](https://mlsanity.com/paper/2008.13064/) (2020) ## Built on by (most cited first, 22 of 22) - [Counterfactual Explanations for Models of Code](https://mlsanity.com/paper/2111.05711/) (2021) - [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023) - [Memorization and Generalization in Neural Code Intelligence Models](https://mlsanity.com/paper/2106.08704/) (2021) - [Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?](https://mlsanity.com/paper/2306.01220/) (2023) - [Graph Neural Networks for Vulnerability Detection: A Counterfactual Explanation](https://mlsanity.com/paper/2404.15687/) (2024) - [Syntax-Guided Program Reduction for Understanding Neural Code Intelligence Models](https://mlsanity.com/paper/2205.14374/) (2022) - [Follow-up Attention: An Empirical Study of Developer and Neural Model Code Exploration](https://mlsanity.com/paper/2210.05506/) (2022) - [Code2Snapshot: Using Code Snapshots for Learning Representations of Source Code](https://mlsanity.com/paper/2111.01097/) (2021) - [Measuring Impacts of Poisoning on Model Parameters and Embeddings for Large Language Models of Code](https://mlsanity.com/paper/2405.11466/) (2024) - [A Critical Study of What Code-LLMs (Do Not) Learn](https://mlsanity.com/paper/2406.11930/) (2024) - [Study of Distractors in Neural Models of Code](https://mlsanity.com/paper/2303.01739/) (2023) - [A Systematic Literature Review on Explainability for Machine/Deep Learning-based Software Engineering Research](https://mlsanity.com/paper/2401.14617/) (2024) - [Is Compression Really Linear with Code Intelligence?](https://mlsanity.com/paper/2505.11441/) (2025) - [Analyzing Latent Concepts in Code Language Models](https://mlsanity.com/paper/2510.00476/) (2025) - [Data-Driven AI Model Signal-Awareness Enhancement and Introspection](https://mlsanity.com/paper/2111.05827/) (2021) - [Redundancy and Concept Analysis for Code-trained Language Models](https://mlsanity.com/paper/2305.00875/) (2023) - [Measuring Impacts of Poisoning on Model Parameters and Neuron Activations: A Case Study of Poisoning CodeBERT](https://mlsanity.com/paper/2402.12936/) (2024) - [Extracting Label-specific Key Input Features for Neural Code Intelligence Models](https://mlsanity.com/paper/2202.06474/) (2022) - [A Study of Variable-Role-based Feature Enrichment in Neural Models of Code](https://mlsanity.com/paper/2303.04942/) (2023) - [Beyond Fidelity: Explaining Vulnerability Localization of Learning-based Detectors](https://mlsanity.com/paper/2401.02686/) (2024) - [LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models](https://mlsanity.com/paper/2505.14759/) (2025) - [From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agents](https://mlsanity.com/paper/2512.13438/) (2025) ## Abstract (arXiv) A wide range of code intelligence (CI) tools, powered by deep neural networks, have been developed recently to improve programming productivity and perform program analysis. To reliably use such tools, developers often need to reason about the behavior of the underlying models and the factors that affect them. This is especially challenging for tools backed by deep neural networks. Various methods have tried to reduce this opacity in the vein of "transparent/interpretable-AI". However, these approaches are often specific to a particular set of network architectures, even requiring access to the network's parameters. This makes them difficult to use for the average programmer, which hinders the reliable adoption of neural CI systems. In this paper, we propose a simple, model-agnostic approach to identify critical input features for models in CI systems, by drawing on software debugging research, specifically delta debugging. Our approach, SIVAND, uses simplification techniques that reduce the size of input programs of a CI model while preserving the predictions of the model. We show that this approach yields remarkably small outputs and is broadly applicable across many model architectures and problem domains. We find that the models in our experiments often rely heavily on just a few syntactic features in input programs. We believe that SIVAND's extracted features may help understand neural CI systems' predictions and learned behavior.