# A Benchmark for Interpretability Methods in Deep Neural Networks

arXiv: https://arxiv.org/abs/1806.10758 · HTML: https://mlsanity.com/paper/1806.10758/

## Key facts

- Cited by 476 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2018.

## Built on by (most cited first, 30 of 476)

- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [FastSHAP: Real-Time Shapley Value Estimation](https://mlsanity.com/paper/2107.07436/) (2021)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation](https://mlsanity.com/paper/2306.07304/) (2023)
- [HIVE: Evaluating the Human Interpretability of Visual Explanations](https://mlsanity.com/paper/2112.03184/) (2021)
- [A Consistent and Efficient Evaluation Strategy for Attribution Methods](https://mlsanity.com/paper/2202.00449/) (2022)
- [Are self-explanations from Large Language Models faithful?](https://mlsanity.com/paper/2401.07927/) (2024)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations](https://mlsanity.com/paper/2106.00786/) (2021)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Improving Deep Learning Interpretability by Saliency Guided Training](https://mlsanity.com/paper/2111.14338/) (2021)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis](https://mlsanity.com/paper/2111.04138/) (2021)
- [Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations](https://mlsanity.com/paper/2103.01890/) (2021)
- [Synthetic Benchmarks for Scientific Research in Explainable Machine Learning](https://mlsanity.com/paper/2106.12543/) (2021)
- [Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation](https://mlsanity.com/paper/2212.04629/) (2022)
- [CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior](https://mlsanity.com/paper/2205.14140/) (2022)
- [Evaluation of post-hoc interpretability methods in time-series classification](https://mlsanity.com/paper/2202.05656/) (2022)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Do Input Gradients Highlight Discriminative Features?](https://mlsanity.com/paper/2102.12781/) (2021)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methods](https://mlsanity.com/paper/2308.06248/) (2023)

## Abstract (arXiv)

We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks. Our results across several large-scale image classification datasets show that many popular interpretability methods produce estimates of feature importance that are not better than a random designation of feature importance. Only certain ensemble based approaches---VarGrad and SmoothGrad-Squared---outperform such a random assignment of importance. The manner of ensembling remains critical, we show that some approaches do no better then the underlying method but carry a far higher computational burden.