# NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networks

arXiv: https://arxiv.org/abs/2203.12915 · HTML: https://mlsanity.com/paper/2203.12915/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing](https://mlsanity.com/paper/1807.10875/) (2018)
- [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018)
- [Concolic Testing for Deep Neural Networks](https://mlsanity.com/paper/1805.00089/) (2018)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [AVA: Adversarial Vignetting Attack against Visual Recognition](https://mlsanity.com/paper/2105.05558/) (2021)
- [Test Selection for Deep Learning Systems](https://mlsanity.com/paper/1904.13195/) (2019)
- [Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertainty](https://mlsanity.com/paper/2004.11573/) (2020)
- [Importance-Driven Deep Learning System Testing](https://mlsanity.com/paper/2002.03433/) (2020)
- [Self-Checking Deep Neural Networks in Deployment](https://mlsanity.com/paper/2103.02371/) (2021)
- [Pasadena: Perceptually Aware and Stealthy Adversarial Denoise Attack](https://mlsanity.com/paper/2007.07097/) (2020)
- [There is Limited Correlation between Coverage and Robustness for Deep Neural Networks](https://mlsanity.com/paper/1911.05904/) (2019)
- [Towards Improved Testing For Deep Learning](https://mlsanity.com/paper/1902.06320/) (2019)

## Built on by (most cited first, 18 of 18)

- [Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization](https://mlsanity.com/paper/2306.02879/) (2023)
- [FAIRER: Fairness as Decision Rationale Alignment](https://mlsanity.com/paper/2306.15299/) (2023)
- [Hierarchical Distribution-Aware Testing of Deep Learning](https://mlsanity.com/paper/2205.08589/) (2022)
- [Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks](https://mlsanity.com/paper/2408.15207/) (2024)
- [A Scenario-Based Functional Testing Approach to Improving DNN Performance](https://mlsanity.com/paper/2307.07083/) (2023)
- [CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space](https://mlsanity.com/paper/2307.09375/) (2023)
- [Neuron Sensitivity Guided Test Case Selection for Deep Learning Testing](https://mlsanity.com/paper/2307.11011/) (2023)
- [Interpretability-Guided Test-Time Adversarial Defense](https://mlsanity.com/paper/2409.15190/) (2024)
- [Feature Map Testing for Deep Neural Networks](https://mlsanity.com/paper/2307.11563/) (2023)
- [IRAD: Implicit Representation-driven Image Resampling against Adversarial Attacks](https://mlsanity.com/paper/2310.11890/) (2023)
- [TraceFL: Interpretability-Driven Debugging in Federated Learning via Neuron Provenance](https://mlsanity.com/paper/2312.13632/) (2023)
- [NeuSemSlice: Towards Effective DNN Model Maintenance via Neuron-level Semantic Slicing](https://mlsanity.com/paper/2407.20281/) (2024)
- [Generalization Beyond Feature Alignment: Concept Activation-Guided Contrastive Learning](https://mlsanity.com/paper/2211.06843/) (2022)
- [Path Analysis for Effective Fault Localization in Deep Neural Networks](https://mlsanity.com/paper/2310.18987/) (2023)
- [Assessing Superposition-Targeted Coverage Criteria for Quantum Neural Networks](https://mlsanity.com/paper/2411.02450/) (2024)
- [Towards Understanding Deep Learning Model in Image Recognition via Coverage Test](https://mlsanity.com/paper/2505.08814/) (2025)
- [Influence-Guided Concolic Testing of Transformer Robustness](https://mlsanity.com/paper/2509.23806/) (2025)
- [ProToken: Token-Level Attribution for Federated Large Language Models](https://mlsanity.com/paper/2601.19672/) (2026)

## Abstract (arXiv)

Deep learning has recently been widely applied to many applications across different domains, e.g., image classification and audio recognition. However, the quality of Deep Neural Networks (DNNs) still raises concerns in the practical operational environment, which calls for systematic testing, especially in safety-critical scenarios. Inspired by software testing, a number of structural coverage criteria are designed and proposed to measure the test adequacy of DNNs. However, due to the blackbox nature of DNN, the existing structural coverage criteria are difficult to interpret, making it hard to understand the underlying principles of these criteria. The relationship between the structural coverage and the decision logic of DNNs is unknown. Moreover, recent studies have further revealed the non-existence of correlation between the structural coverage and DNN defect detection, which further posts concerns on what a suitable DNN testing criterion should be. In this paper, we propose the interpretable coverage criteria through constructing the decision structure of a DNN. Mirroring the control flow graph of the traditional program, we first extract a decision graph from a DNN based on its interpretation, where a path of the decision graph represents a decision logic of the DNN. Based on the control flow and data flow of the decision graph, we propose two variants of path coverage to measure the adequacy of the test cases in exercising the decision logic. The higher the path coverage, the more diverse decision logic the DNN is expected to be explored. Our large-scale evaluation results demonstrate that: the path in the decision graph is effective in characterizing the decision of the DNN, and the proposed coverage criteria are also sensitive with errors including natural errors and adversarial examples, and strongly correlated with the output impartiality.