# A Human-Grounded Evaluation of SHAP for Alert Processing

arXiv: https://arxiv.org/abs/1907.03324 · HTML: https://mlsanity.com/paper/1907.03324/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2019.

## Built on by (most cited first, 23 of 23)

- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications](https://mlsanity.com/paper/2103.04244/) (2021)
- [Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making](https://mlsanity.com/paper/2101.05303/) (2021)
- [In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making](https://mlsanity.com/paper/2305.07722/) (2023)
- [Selective Explanations: Leveraging Human Input to Align Explainable AI](https://mlsanity.com/paper/2301.09656/) (2023)
- [Machine Explanations and Human Understanding](https://mlsanity.com/paper/2202.04092/) (2022)
- [As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making](https://mlsanity.com/paper/2501.12868/) (2025)
- [A Fine-grained Interpretability Evaluation Benchmark for Neural NLP](https://mlsanity.com/paper/2205.11097/) (2022)
- [Impact Of Explainable AI On Cognitive Load: Insights From An Empirical Study](https://mlsanity.com/paper/2304.08861/) (2023)
- [Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-making](https://mlsanity.com/paper/2204.05030/) (2022)
- [Using AI Uncertainty Quantification to Improve Human Decision-Making](https://mlsanity.com/paper/2309.10852/) (2023)
- [Developing a Fidelity Evaluation Approach for Interpretable Machine Learning](https://mlsanity.com/paper/2106.08492/) (2021)
- [On the overlooked issue of defining explanation objectives for local-surrogate explainers](https://mlsanity.com/paper/2106.05810/) (2021)
- [Machine Learning for Fraud Detection in E-Commerce: A Research Agenda](https://mlsanity.com/paper/2107.01979/) (2021)
- [Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts](https://mlsanity.com/paper/2205.05749/) (2022)
- [An Interpretability Evaluation Benchmark for Pre-trained Language Models](https://mlsanity.com/paper/2207.13948/) (2022)
- [Explainable and Human-Grounded AI for Decision Support Systems: The Theory of Epistemic Quasi-Partnerships](https://mlsanity.com/paper/2409.14839/) (2024)
- [OpenHEXAI: An Open-Source Framework for Human-Centered Evaluation of Explainable Machine Learning](https://mlsanity.com/paper/2403.05565/) (2024)
- [Selective Explanations](https://mlsanity.com/paper/2405.19562/) (2024)
- [T-Explainer: A Model-Agnostic Explainability Framework Based on Gradients](https://mlsanity.com/paper/2404.16495/) (2024)
- [Data-Driven Evaluation of Training Action Space for Reinforcement Learning](https://mlsanity.com/paper/2204.03840/) (2022)
- [Exploring How Anomalous Model Input and Output Alerts Affect Decision-Making in Healthcare](https://mlsanity.com/paper/2204.13194/) (2022)

## Abstract (arXiv)

In the past years, many new explanation methods have been proposed to achieve interpretability of machine learning predictions. However, the utility of these methods in practical applications has not been researched extensively. In this paper we present the results of a human-grounded evaluation of SHAP, an explanation method that has been well-received in the XAI and related communities. In particular, we study whether this local model-agnostic explanation method can be useful for real human domain experts to assess the correctness of positive predictions, i.e. alerts generated by a classifier. We performed experimentation with three different groups of participants (159 in total), who had basic knowledge of explainable machine learning. We performed a qualitative analysis of recorded reflections of experiment participants performing alert processing with and without SHAP information. The results suggest that the SHAP explanations do impact the decision-making process, although the model's confidence score remains to be a leading source of evidence. We statistically test whether there is a significant difference in task utility metrics between tasks for which an explanation was available and tasks in which it was not provided. As opposed to common intuitions, we did not find a significant difference in alert processing performance when a SHAP explanation is available compared to when it is not.