mlsanityData to 7 Oct 2026

Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding

arXiv 2603.25251 ↗First posted 26 Mar 2026cs.HC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Explainable AI (XAI) methods are commonly evaluated using functional correctness metrics, sometimes termed faithfulness or fidelity, which estimate how closely an explanation reflects the model's reasoning. Higher correctness is assumed to produce better human understanding, but this link has not been tested with controlled levels. We conducted a user study (N=200) that manipulated explanation correctness at four levels (100%, 85%, 70%, 55%) in a synthetic time series classification task where participants could not rely on domain knowledge or visual intuition. Correctness was defined against a known ground truth, not estimated from a trained model. Participants predicted a simulated AI's decisions from feature-attribution-style explanations (forward simulation), and we used their forward simulation accuracy as a proxy for understanding. Correctness affected understanding, but not at every level: forward simulation accuracy dropped at 70% and 55% relative to fully correct explanations, with no further reduction below 70% and no conclusive difference between 85% and 100%. Lower correctness reduced how many participants predicted accurately rather than lowering accuracy uniformly, and even fully correct explanations did not guarantee understanding: forward simulation accuracy was bimodal there, with only a subset of participants well above chance. In exploratory analyses, some participants predicted decisions accurately but described the wrong pattern when asked what the AI used, and self-reported ratings correlated with forward simulation accuracy only when explanations were fully correct. These findings show that not all differences in functional correctness translate to differences in human understanding, highlighting the need to validate functional metrics against human outcomes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 73 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

49 of 73 references.

References · 49

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  5. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  6. 2018Sanity Checks for Saliency Mapscited by 1,149
  7. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  8. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  9. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  10. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  11. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  12. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
37 more
  1. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  2. 2017The (Un)reliability of saliency methodscited by 331
  3. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  4. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  5. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  6. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  7. 2020Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAIcited by 125
  8. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  9. 2021What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Researchcited by 119
  10. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  11. 2020Restricting the Flow: Information Bottlenecks for Attributioncited by 110
  12. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  13. 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
  14. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  15. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  16. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  17. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  18. 2019Towards a Rigorous Evaluation of XAI Methods on Time Seriescited by 67
  19. 2022Evaluation of post-hoc interpretability methods in time-series classificationcited by 53
  20. 2019How model accuracy and explanation fidelity influence user trustcited by 44
  21. 2023The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantuscited by 37
  22. 2024Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarkscited by 28
  23. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  24. 2021Focus! Rating XAI Methods and Finding Biasescited by 20
  25. 2022Machine Explanations and Human Understandingcited by 20
  26. 2023Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanomacited by 18
  27. 2023A Deep Dive into Perturbations as Evaluation Technique for Time Series XAIcited by 11
  28. 2024Contrastive Explanations That Anticipate Human Misconceptions Can Improve Human Decision-Making Skillscited by 11
  29. 2024Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaborationcited by 8
  30. 2025Evaluating Model Explanations without Ground Truthcited by 6
  31. 2025Class-Dependent Perturbation Effects in Evaluating Time Series Attributionscited by 5
  32. 2025Fewer Than 1% of Explainable AI Papers Validate Explainability with Humanscited by 4
  33. 2025Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency mapscited by 4
  34. 2024Dermatologist-like explainable AI enhances melanoma diagnosis accuracy: eye-tracking studycited by 2
  35. 2025Faithful Counterfactual Visual Explanations (FCVE)cited by 2
  36. 2025Why Do Class-Dependent Evaluation Effects Occur with Time Series Feature Attributions? A Synthetic Data Investigationcited by 2
  37. 2026xaitimesynth: A Python Package for Evaluating Attribution Methods for Time Series with Synthetic Ground Truthcited by 1

Built on by

Not cited yet.