mlsanityData to 7 Oct 2026

From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation

arXiv 2412.05592 ↗First posted 7 Dec 2024cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

The lack of ground truth explanation labels is a fundamental challenge for quantitative evaluation in explainable artificial intelligence (XAI). This challenge becomes especially problematic when evaluation methods have numerous hyperparameters that must be specified by the user, as there is no ground truth to determine an optimal hyperparameter selection. It is typically not feasible to do an exhaustive search of hyperparameters so researchers typically make a normative choice based on similar studies in the literature, which provides great flexibility for the user. In this work, we illustrate how this flexibility can be exploited to manipulate the evaluation outcome. We frame this manipulation as an adversarial attack on the evaluation where seemingly innocent changes in hyperparameter setting significantly influence the evaluation outcome. We demonstrate the effectiveness of our manipulation across several datasets with large changes in evaluation outcomes across several explanation methods and models. Lastly, we propose a mitigation strategy based on ranking across hyperparameters that aims to provide robustness towards such manipulation. This work highlights the difficulty of conducting reliable XAI evaluation and emphasizes the importance of a holistic and transparent approach to evaluation in XAI.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 7 Dec 2024.

Builds on

35 of 55 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2018Sanity Checks for Saliency Mapscited by 1,149
  6. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  7. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  8. 2017One pixel attack for fooling deep neural networkscited by 579
  9. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  10. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  11. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  12. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
23 more
  1. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 342
  2. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
  3. 2020Evaluating and Aggregating Feature-based Model Explanationscited by 185
  4. 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
  5. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 156
  6. 2020Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAIcited by 125
  7. 2022OpenXAI: Towards a Transparent Evaluation of Model Explanationscited by 125
  8. 2019Sanity Checks for Saliency Metricscited by 119
  9. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 100
  10. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  11. 2020On quantitative aspects of model interpretabilitycited by 71
  12. 2022Framework for Evaluating Faithfulness of Local Explanationscited by 55
  13. 2020IROF: a low resource evaluation metric for explanation methodscited by 45
  14. 2023The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantuscited by 37
  15. 2021NoiseGrad: Enhancing Explanations by Introducing Stochasticity to Model Weightscited by 28
  16. 2024Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarkscited by 28
  17. 2022Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanationscited by 22
  18. 2022On the Relationship Between Explanation and Prediction: A Causal Viewcited by 14
  19. 2021RELAX: Representation Learning Explainabilitycited by 12
  20. 2018A Note about: Local Explanation Methods for Deep Neural Networks lack Sensitivity to Parameter Valuescited by 9
  21. 2024A Fresh Look at Sanity Checks for Saliency Mapscited by 8
  22. 2021Evaluating Input Perturbation Methods for Interpreting CNNs and Saliency Map Comparisoncited by 7
  23. 2024Toward Understanding the Disagreement Problem in Neural Network Feature Attributioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Bridging the Gap in XAI-Why Reliable Metrics Matter for Explainability and Compliancecited by 6
  2. 2025The Role of Hyperparameters in Predictive Multiplicitycited by 4
  3. 2026Position: Explainable AI is Causality in Disguisecited by 0
  4. 2026Mask-Induced Displacement in Audio XAI via Logit Trajectory Decompositioncited by 0