mlsanityData to 7 Oct 2026

A Benchmark for Interpretability Methods in Deep Neural Networks

arXiv 1806.10758 ↗First posted 28 Jun 2018cs.LG cs.AI stat.MLcited by 467

Not on the map. Why?

Abstract (arXiv)

We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks. Our results across several large-scale image classification datasets show that many popular interpretability methods produce estimates of feature importance that are not better than a random designation of feature importance. Only certain ensemble based approaches---VarGrad and SmoothGrad-Squared---outperform such a random assignment of importance. The manner of ensembling remains critical, we show that some approaches do no better then the underlying method but carry a far higher computational burden.

Key facts. Cited by 467 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 72; 2022: 89; 2023: 79; 2024: 89; 2025: 55; 2026: 83.

467 papers.

Sort:

By area:

  1. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  2. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  3. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  4. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  5. 2021FastSHAP: Real-Time Shapley Value Estimationcited by 123
  6. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  7. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  8. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  9. 2024Are self-explanations from Large Language Models faithful?cited by 101
  10. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 100
  11. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  12. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
38 more
  1. 2023Concept-based Explainable Artificial Intelligence: A Surveycited by 97
  2. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  3. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  4. 2022Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanationscited by 77
  5. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  6. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  7. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  8. 2021Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysiscited by 65
  9. 2022TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversationscited by 65
  10. 2021Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretationscited by 59
  11. 2021Synthetic Benchmarks for Scientific Research in Explainable Machine Learningcited by 59
  12. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  13. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  14. 2022Evaluation of post-hoc interpretability methods in time-series classificationcited by 53
  15. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  16. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
  17. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  18. 2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 49
  19. 2021Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networkscited by 48
  20. 2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
  21. 2022Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capabilitycited by 46
  22. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  23. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  24. 2025Vision Transformers Don't Need Trained Registerscited by 39
  25. 2022Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysiscited by 38
  26. 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
  27. 2021Explaining a Series of Models by Propagating Shapley Valuescited by 37
  28. 2022Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networkscited by 37
  29. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  30. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  31. 2022Missingness Bias in Model Debuggingcited by 36
  32. 2021Zorro: Valid, Sparse, and Stable Explanations in Graph Neural Networkscited by 34
  33. 2022Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysiscited by 34
  34. 2024Gradient based Feature Attribution in Explainable AI: A Technical Reviewcited by 34
  35. 2022Making Sense of Dependence: Efficient Black-box Explanations Using Dependence Measurecited by 33
  36. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  37. 2021Neural Response Interpretation through the Lens of Critical Pathwayscited by 32
  38. 2021Interpretable Semantic Photo Geolocationcited by 32

50 most cited of 467 All 467 (JSON)