# Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networks

arXiv: https://arxiv.org/abs/2303.01538 · HTML: https://mlsanity.com/paper/2303.01538/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Evaluating and Aggregating Feature-based Model Explanations](https://mlsanity.com/paper/2005.00631/) (2020)
- [XRAI: Better Attributions Through Regions](https://mlsanity.com/paper/1906.02825/) (2019)
- [A Consistent and Efficient Evaluation Strategy for Attribution Methods](https://mlsanity.com/paper/2202.00449/) (2022)
- [Learning how to explain neural networks: PatternNet and PatternAttribution](https://mlsanity.com/paper/1705.05598/) (2017)
- [On quantitative aspects of model interpretability](https://mlsanity.com/paper/2007.07584/) (2020)
- [Framework for Evaluating Faithfulness of Local Explanations](https://mlsanity.com/paper/2202.00734/) (2022)
- [IROF: a low resource evaluation metric for explanation methods](https://mlsanity.com/paper/2003.08747/) (2020)
- [Circumventing Outliers of AutoAugment with Knowledge Distillation](https://mlsanity.com/paper/2003.11342/) (2020)
- [Enforcing Interpretability and its Statistical Impacts: Trade-offs between Accuracy and Interpretability](https://mlsanity.com/paper/2010.13764/) (2020)
- [Evaluation of importance estimators in deep learning classifiers for Computed Tomography](https://mlsanity.com/paper/2209.15398/) (2022)
- [Input Bias in Rectified Gradients and Modified Saliency Maps](https://mlsanity.com/paper/2011.05002/) (2020)

## Built on by (most cited first, 9 of 9)

- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks](https://mlsanity.com/paper/2401.06654/) (2024)
- [Class-Discriminative Attention Maps for Vision Transformers](https://mlsanity.com/paper/2312.02364/) (2023)
- [SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation](https://mlsanity.com/paper/2411.01710/) (2024)
- [False Sense of Security in Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2405.03820/) (2024)
- [Rethinking Explanation Evaluation under the Retraining Scheme](https://mlsanity.com/paper/2511.08281/) (2025)
- [Integrative CAM: Adaptive Layer Fusion for Comprehensive Interpretation of CNNs](https://mlsanity.com/paper/2412.01354/) (2024)
- [Explain and Monitor Deep Learning Models for Computer Vision using Obz AI](https://mlsanity.com/paper/2508.18188/) (2025)
- [Regularizing Attention Scores with Bootstrapping](https://mlsanity.com/paper/2604.01339/) (2026)

## Abstract (arXiv)

Post-hoc explanation methods attempt to make the inner workings of deep neural networks more interpretable. However, since a ground truth is in general lacking, local post-hoc interpretability methods, which assign importance scores to input features, are challenging to evaluate. One of the most popular evaluation frameworks is to perturb features deemed important by an interpretability method and to measure the change in prediction accuracy. Intuitively, a large decrease in prediction accuracy would indicate that the explanation has correctly quantified the importance of features with respect to the prediction outcome (e.g., logits). However, the change in the prediction outcome may stem from perturbation artifacts, since perturbed samples in the test dataset are out of distribution (OOD) compared to the training dataset and can therefore potentially disturb the model in an unexpected manner. To overcome this challenge, we propose feature perturbation augmentation (FPA) which creates and adds perturbed images during the model training. Through extensive computational experiments, we demonstrate that FPA makes deep neural networks (DNNs) more robust against perturbations. Furthermore, training DNNs with FPA demonstrate that the sign of importance scores may explain the model more meaningfully than has previously been assumed. Overall, FPA is an intuitive data augmentation technique that improves the evaluation of post-hoc interpretability methods.