mlsanityData to 14 Oct 2026

A Unified Study of Machine Learning Explanation Evaluation Metrics

arXiv 2203.14265 ↗First posted 27 Mar 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

The growing need for trustworthy machine learning has led to the blossom of interpretability research. Numerous explanation methods have been developed to serve this purpose. However, these methods are deficiently and inappropriately evaluated. Many existing metrics for explanations are introduced by researchers as by-products of their proposed explanation techniques to demonstrate the advantages of their methods. Although widely used, they are more or less accused of problems. We claim that the lack of acknowledged and justified metrics results in chaos in benchmarking these explanation methods -- Do we really have good/bad explanation when a metric gives a high/low score? We split existing metrics into two categories and demonstrate that they are insufficient to properly evaluate explanations for multiple reasons. We propose guidelines in dealing with the problems in evaluating machine learning explanation and encourage researchers to carefully deal with these problems when developing explanation techniques and metrics.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 27 Mar 2022.

Builds on

41 of 55 references.

References · 41

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  12. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
29 more
  1. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  2. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  3. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  4. 2018Sanity Checks for Saliency Mapscited by 1,156
  5. 2020Concept Bottleneck Modelscited by 1,032
  6. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  7. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  8. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  9. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  10. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  11. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  12. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  13. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  14. 2016Top-down Neural Attention by Excitation Backpropcited by 321
  15. 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 319
  16. 2019On the (In)fidelity and Sensitivity for Explanationscited by 310
  17. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  18. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 259
  19. 2020Debugging Tests for Model Explanationscited by 147
  20. 2019Sanity Checks for Saliency Metricscited by 119
  21. 2018L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Datacited by 100
  22. 2019Benchmarking Attribution Methods with Relative Feature Importancecited by 89
  23. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  24. 2018Stakeholders in Explainable AIcited by 48
  25. 2016Explaining Predictions of Non-Linear Classifiers in NLPcited by 37
  26. 2020OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samplescited by 33
  27. 2019Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithmscited by 32
  28. 2020A Framework to Learn with Interpretationcited by 23
  29. 2021Self-Interpretable Model with TransformationEquivariant Interpretationcited by 21

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 2; 2025: 1.

6 papers.

  1. 2023The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantuscited by 37
  2. 2024On the Faithfulness of Vision Transformer Explanationscited by 19
  3. 2024On the Evaluation Consistency of Attribution-based Explanationscited by 5
  4. 2025Absolute Evaluation Measures for Machine Learning: A Surveycited by 2
  5. 2022Fidelity of Ensemble Aggregation for Saliency Map Explanations using Bayesian Optimization Techniquescited by 1
  6. 2022HEX: Human-in-the-loop Explainability via Deep Reinforcement Learningcited by 0