# SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretability

arXiv: https://arxiv.org/abs/2208.09418 · HTML: https://mlsanity.com/paper/2208.09418/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 19 Aug 2022.

## Builds on (most cited first, 30 of 35)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Robustness via curvature regularization, and vice versa](https://mlsanity.com/paper/1811.09716/) (2018)
- [Explainable artificial intelligence (XAI) in deep learning-based medical image analysis](https://mlsanity.com/paper/2107.10912/) (2021)
- [Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI](https://mlsanity.com/paper/2003.07258/) (2020)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [Fooling Neural Network Interpretations via Adversarial Model Manipulation](https://mlsanity.com/paper/1902.02041/) (2019)
- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [Interpretable Deep Learning under Fire](https://mlsanity.com/paper/1812.00891/) (2018)
- [Synthetic Benchmarks for Scientific Research in Explainable Machine Learning](https://mlsanity.com/paper/2106.12543/) (2021)
- [Framework for Evaluating Faithfulness of Local Explanations](https://mlsanity.com/paper/2202.00734/) (2022)
- [A Statistical Approach to Assessing Neural Network Robustness](https://mlsanity.com/paper/1811.07209/) (2018)
- [BayLIME: Bayesian Local Interpretable Model-Agnostic Explanations](https://mlsanity.com/paper/2012.03058/) (2020)
- [Towards Robust Explanations for Deep Neural Networks](https://mlsanity.com/paper/2012.10425/) (2020)
- [Enhancing Adversarial Training with Second-Order Statistics of Weights](https://mlsanity.com/paper/2203.06020/) (2022)
- [Interpreting and Evaluating Neural Network Robustness](https://mlsanity.com/paper/1905.04270/) (2019)
- [Statistically Robust Neural Network Classification](https://mlsanity.com/paper/1912.04884/) (2019)

## Built on by (most cited first, 28 of 28)

- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Adversarial attacks and defenses in explainable artificial intelligence: A survey](https://mlsanity.com/paper/2306.06123/) (2023)
- [Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance](https://mlsanity.com/paper/2304.06715/) (2023)
- [ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion Models against Stochastic Perturbation](https://mlsanity.com/paper/2402.15429/) (2024)
- [TrajPAC: Towards Robustness Verification of Pedestrian Trajectory Prediction Models](https://mlsanity.com/paper/2308.05985/) (2023)
- [What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systems](https://mlsanity.com/paper/2307.11784/) (2023)
- [Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey](https://mlsanity.com/paper/2409.18214/) (2024)
- [Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide](https://mlsanity.com/paper/2502.14833/) (2025)
- [Out-of-Bounding-Box Triggers: A Stealthy Approach to Cheat Object Detectors](https://mlsanity.com/paper/2410.10091/) (2024)
- [A comprehensive interpretable machine learning framework for Mild Cognitive Impairment and Alzheimer's disease diagnosis](https://mlsanity.com/paper/2412.09376/) (2024)
- [Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning](https://mlsanity.com/paper/2503.08636/) (2025)
- [Evolutionary approaches to explainable machine learning](https://mlsanity.com/paper/2306.14786/) (2023)
- [Unsupervised discovery of Interpretable Visual Concepts](https://mlsanity.com/paper/2309.00018/) (2023)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Evolutionary Computation and Explainable AI: A Roadmap to Understandable Intelligent Systems](https://mlsanity.com/paper/2406.07811/) (2024)
- [On the Robustness of Global Feature Effect Explanations](https://mlsanity.com/paper/2406.09069/) (2024)
- [Scene-Aware Explainable Multimodal Trajectory Prediction](https://mlsanity.com/paper/2410.16795/) (2024)
- [Robust Explanations Through Uncertainty Decomposition: A Path to Trustworthier AI](https://mlsanity.com/paper/2507.12913/) (2025)
- [SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddings](https://mlsanity.com/paper/2606.29623/) (2026)
- [XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensics](https://mlsanity.com/paper/2402.02452/) (2024)
- [XAI-Units: Benchmarking Explainability Methods with Unit Tests](https://mlsanity.com/paper/2506.01059/) (2025)
- [PRBench: A Standardized Probabilistic Robustness Benchmark](https://mlsanity.com/paper/2511.01724/) (2025)
- [Non-Parametric Probabilistic Robustness: A Conservative Risk Estimator under Unknown Perturbation Distributions](https://mlsanity.com/paper/2511.17380/) (2025)
- [Quantifying Fidelity: A Decisive Feature Approach to Comparing Synthetic and Real Imagery](https://mlsanity.com/paper/2512.16468/) (2025)
- [Are Classification Robustness and Explanation Robustness Really Strongly Correlated? An Analysis Through Input Loss Landscape](https://mlsanity.com/paper/2403.06013/) (2024)
- [Revealing Vulnerabilities of Neural Networks in Parameter Learning and Defense Against Explanation-Aware Backdoors](https://mlsanity.com/paper/2403.16569/) (2024)
- [Beyond the Veil of Similarity: Quantifying Semantic Continuity in Explainable AI](https://mlsanity.com/paper/2407.12950/) (2024)
- [EVO-LRP: Evolutionary Optimization of LRP for Interpretable Model Explanations](https://mlsanity.com/paper/2509.23585/) (2025)

## Abstract (arXiv)

Interpretability of Deep Learning (DL) is a barrier to trustworthy AI. Despite great efforts made by the Explainable AI (XAI) community, explanations lack robustness -- indistinguishable input perturbations may lead to different XAI results. Thus, it is vital to assess how robust DL interpretability is, given an XAI method. In this paper, we identify several challenges that the state-of-the-art is unable to cope with collectively: i) existing metrics are not comprehensive; ii) XAI techniques are highly heterogeneous; iii) misinterpretations are normally rare events. To tackle these challenges, we introduce two black-box evaluation methods, concerning the worst-case interpretation discrepancy and a probabilistic notion of how robust in general, respectively. Genetic Algorithm (GA) with bespoke fitness function is used to solve constrained optimisation for efficient worst-case evaluation. Subset Simulation (SS), dedicated to estimate rare event probabilities, is used for evaluating overall robustness. Experiments show that the accuracy, sensitivity, and efficiency of our methods outperform the state-of-the-arts. Finally, we demonstrate two applications of our methods: ranking robust XAI methods and selecting training schemes to improve both classification and interpretation robustness.