mlsanityData to 14 Oct 2026

Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysis

arXiv 2309.11987 ↗First posted 21 Sep 2023cs.LG cs.AI cs.HCcited by 4

Not on the map. Why?

Abstract (arXiv)

Post-hoc explainability methods aim to clarify predictions of black-box machine learning models. However, it is still largely unclear how well users comprehend the provided explanations and whether these increase the users ability to predict the model behavior. We approach this question by conducting a user study to evaluate comprehensibility and predictability in two widely used tools: LIME and SHAP. Moreover, we investigate the effect of counterfactual explanations and misclassifications on users ability to understand and predict the model behavior. We find that the comprehensibility of SHAP is significantly reduced when explanations are provided for samples near a model's decision boundary. Furthermore, we find that counterfactual explanations and misclassifications can significantly increase the users understanding of how a machine learning model is making decisions. Based on our findings, we also derive design recommendations for future post-hoc explainability methods with increased comprehensibility and predictability.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 18 of its 39 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

18 of 39 references.

References · 18

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  4. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  5. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  6. 2020Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIcited by 167
  7. 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 166
  8. 2019What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Usecited by 135
  9. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  10. 2020Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challengescited by 120
  11. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  12. 2019Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approachescited by 101
6 more
  1. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  2. 2019explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learningcited by 70
  3. 2019Quantifying Interpretability and Trust in Machine Learning Systemscited by 49
  4. 2019How model accuracy and explanation fidelity influence user trustcited by 45
  5. 2018A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learningcited by 21
  6. 2020ALEX: Active Learning based Enhancement of a Model's Explainabilitycited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Do Cognitively Interpretable Reasoning Traces Improve LLM Performance?cited by 11
  2. 2025Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillationcited by 8
  3. 2025Explainable Reinforcement Learning for Formula One Race Strategycited by 1
  4. 2024Dataset resulting from the user study on comprehensibility of explainable AI algorithmscited by 0