# An empirical study of the effect of background data size on the stability of SHapley Additive exPlanations (SHAP) for deep learning models

arXiv: https://arxiv.org/abs/2204.11351 · HTML: https://mlsanity.com/paper/2204.11351/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 24 Apr 2022.

## Builds on (most cited first, 6 of 6)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [Robust and Stable Black Box Explanations](https://mlsanity.com/paper/2011.06169/) (2020)
- [Explainable artificial intelligence model to predict acute critical illness from electronic health records](https://mlsanity.com/paper/1912.01266/) (2019)
- [Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies](https://mlsanity.com/paper/2107.09951/) (2021)
- [AutoScore-Imbalance: An interpretable machine learning tool for development of clinical scores with rare events data](https://mlsanity.com/paper/2107.06039/) (2021)

## Built on by (most cited first, 4 of 4)

- [Explanation Multiplicity in SHAP: Characterization and Assessment](https://mlsanity.com/paper/2601.12654/) (2026)
- [Streamlining models with explanations in the learning loop](https://mlsanity.com/paper/2302.07760/) (2023)
- [Foundation Model Makes Clustering A Better Initialization For Cold-Start Active Learning](https://mlsanity.com/paper/2402.02561/) (2024)
- [Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition](https://mlsanity.com/paper/2412.16003/) (2024)

## Abstract (arXiv)

Nowadays, the interpretation of why a machine learning (ML) model makes certain inferences is as crucial as the accuracy of such inferences. Some ML models like the decision tree possess inherent interpretability that can be directly comprehended by humans. Others like artificial neural networks (ANN), however, rely on external methods to uncover the deduction mechanism. SHapley Additive exPlanations (SHAP) is one of such external methods, which requires a background dataset when interpreting ANNs. Generally, a background dataset consists of instances randomly sampled from the training dataset. However, the sampling size and its effect on SHAP remain to be unexplored. In our empirical study on the MIMIC-III dataset, we show that the two core explanations - SHAP values and variable rankings fluctuate when using different background datasets acquired from random sampling, indicating that users cannot unquestioningly trust the one-shot interpretation from SHAP. Luckily, such fluctuation decreases with the increase of the background dataset size. Also, we notice an U-shape in the stability assessment of SHAP variable rankings, demonstrating that SHAP is more reliable in ranking the most and least important variables compared to moderately important ones. Overall, our results suggest that users should take into account how background data affects SHAP results, with improved SHAP stability as the background sample size increases.