# Feature-Guided Analysis of Neural Networks: A Replication Study

arXiv: https://arxiv.org/abs/2511.00052 · HTML: https://mlsanity.com/paper/2511.00052/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 27 of 27)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Scaled-YOLOv4: Scaling Cross Stage Partial Network](https://mlsanity.com/paper/2011.08036/) (2020)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [Concept Activation Regions: A Generalized Framework For Concept-Based Explanations](https://mlsanity.com/paper/2209.11222/) (2022)
- [Software Engineering Challenges of Deep Learning](https://mlsanity.com/paper/1810.12034/) (2018)
- [A Survey on Deep Learning for Software Engineering](https://mlsanity.com/paper/2011.14597/) (2020)
- [DeepLocalize: Fault Localization for Deep Neural Networks](https://mlsanity.com/paper/2103.03376/) (2021)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [DeepFault: Fault Localization for Deep Neural Networks](https://mlsanity.com/paper/1902.05974/) (2019)
- [Arachne: Search Based Repair of Deep Neural Networks](https://mlsanity.com/paper/1912.12463/) (2019)
- [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021)
- [DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs](https://mlsanity.com/paper/2205.01938/) (2022)
- [Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models](https://mlsanity.com/paper/2403.09906/) (2024)
- [Can GPT-4 Replicate Empirical Software Engineering Research?](https://mlsanity.com/paper/2310.01727/) (2023)
- [Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study](https://mlsanity.com/paper/2401.03069/) (2024)
- [Mutation-based Fault Localization of Deep Neural Networks](https://mlsanity.com/paper/2309.05067/) (2023)
- [An Empirical Study on LLM-based Classification of Requirements-related Provisions in Food-safety Regulations](https://mlsanity.com/paper/2501.14683/) (2025)
- [Open Science in Software Engineering](https://mlsanity.com/paper/1904.06499/) (2019)
- [Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study](https://mlsanity.com/paper/2403.13729/) (2024)
- [Prophecy: Inferring Formal Properties from Neuron Activations](https://mlsanity.com/paper/2509.21677/) (2025)
- [Replication studies considered harmful](https://mlsanity.com/paper/1802.04580/) (2018)
- [Practical Guidelines for the Selection and Evaluation of Natural Language Processing Techniques in Requirements Engineering](https://mlsanity.com/paper/2401.01508/) (2024)

## Built on by (most cited first, 2 of 2)

- [Ensembles-based Feature Guided Analysis](https://mlsanity.com/paper/2603.19653/) (2026)
- [Engineering Resource-constrained Software Systems with DNN Components: a Concept-based Pruning Approach](https://mlsanity.com/paper/2604.09988/) (2026)

## Abstract (arXiv)

Understanding why neural networks make certain decisions is pivotal for their use in safety-critical applications. Feature-Guided Analysis (FGA) extracts slices of neural networks relevant to their tasks. Existing feature-guided approaches typically monitor the activation of the neural network neurons to extract the relevant rules. Preliminary results are encouraging and demonstrate the feasibility of this solution by assessing the precision and recall of Feature-Guided Analysis on two pilot case studies. However, the applicability in industrial contexts needs additional empirical evidence. To mitigate this need, this paper assesses the applicability of FGA on a benchmark made by the MNIST and LSC datasets. We assessed the effectiveness of FGA in computing rules that explain the behavior of the neural network. Our results show that FGA has a higher precision on our benchmark than the results from the literature. We also evaluated how the selection of the neural network architecture, training, and feature selection affect the effectiveness of FGA. Our results show that the selection significantly affects the recall of FGA, while it has a negligible impact on its precision.