mlsanityData to 7 Oct 2026

The Solvability of Interpretability Evaluation Metrics

arXiv 2205.08696 ↗First posted 18 May 2022cs.LG cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Feature attribution methods are popular for explaining neural network predictions, and they are often evaluated on metrics such as comprehensiveness and sufficiency. In this paper, we highlight an intriguing property of these metrics: their solvability. Concretely, we can define the problem of optimizing an explanation for a metric, which can be solved by beam search. This observation leads to the obvious yet unaddressed question: why do we use explainers (e.g., LIME) not based on solving the target metric, if the metric value represents explanation quality? We present a series of investigations showing strong performance of this beam search explainer and discuss its broader implication: a definition-evaluation duality of interpretability concepts. We implement the explainer and release the Python solvex package for models of text, image and tabular domains.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

41 of 53 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  9. 2018Sanity Checks for Saliency Mapscited by 1,149
  10. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  11. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  12. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
29 more
  1. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  2. 2017Interpretation of Neural Networks is Fragilecited by 443
  3. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
  4. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  5. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  6. 2019Is Attention Interpretable?cited by 387
  7. 2018On the Robustness of Interpretability Methodscited by 310
  8. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  9. 2016Understanding Neural Networks through Representation Erasurecited by 223
  10. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  11. 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
  12. 2020Debugging Tests for Model Explanationscited by 146
  13. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  14. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  15. 2016Yelp Dataset Challenge: Review Rating Predictioncited by 84
  16. 2018Model Agnostic Supervised Local Explanationscited by 83
  17. 2019AllenNLP Interpret: A Framework for Explaining Predictions of NLP Modelscited by 74
  18. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  19. 2018A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizationscited by 59
  20. 2020Interpretation of NLP models through input marginalizationcited by 43
  21. 2022A Comparative Study of Faithfulness Metrics for Model Interpretability Methodscited by 38
  22. 2021Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classificationcited by 29
  23. 2021The Role of Explainability in Assuring Safety of Machine Learning in Healthcarecited by 26
  24. 2019Leveraging Latent Features for Local Explanationscited by 21
  25. 2021Logic Traps in Evaluating Attribution Scorescited by 18
  26. 2021The Irrationality of Neural Rationale Modelscited by 16
  27. 2022ExSum: From Local Explanations to Model Understandingcited by 15
  28. 2019Towards a Unified Evaluation of Explanation Methods without Ground Truthcited by 6
  29. 2021Double Trouble: How to not explain a text classifier's decisions using counterfactuals synthesized by masked language models?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 2; 2025: 2; 2026: 2.

11 papers.

  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  2. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  3. 2022Impossibility Theorems for Feature Attributioncited by 78
  4. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  5. 2023Stability Guarantees for Feature Attributions with Multiplicative Smoothingcited by 14
  6. 2024Interpretability Needs a New Paradigmcited by 10
  7. 2024Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainabilitycited by 10
  8. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  9. 2023Faithfulness Measurable Masked Language Modelscited by 5
  10. 2026From Mechanistic to Compositional Interpretabilitycited by 1
  11. 2026Tree-of-Evidence: Efficient "System 2" Search for Faithful Multimodal Groundingcited by 0