# DeltaSHAP: Explaining Prediction Evolutions in Online Patient Monitoring with Shapley Values

arXiv: https://arxiv.org/abs/2507.02342 · HTML: https://mlsanity.com/paper/2507.02342/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 21 of 21)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Set Functions for Time Series](https://mlsanity.com/paper/1909.12064/) (2019)
- [Sampling Permutations for Shapley Value Estimation](https://mlsanity.com/paper/2104.12199/) (2021)
- [Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series](https://mlsanity.com/paper/2107.14293/) (2021)
- [TimeSHAP: Explaining Recurrent Models through Sequence Perturbations](https://mlsanity.com/paper/2012.00073/) (2020)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [Clinical Intervention Prediction and Understanding using Deep Networks](https://mlsanity.com/paper/1705.08498/) (2017)
- [TimeX++: Learning Time-Series Explanations with Information Bottleneck](https://mlsanity.com/paper/2405.09308/) (2024)
- [Temporal Dependencies in Feature Importance for Time Series Predictions](https://mlsanity.com/paper/2107.14317/) (2021)
- [Learning Perturbations to Explain Time Series Predictions](https://mlsanity.com/paper/2305.18840/) (2023)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [TIMING: Temporality-Aware Integrated Gradients for Time Series Explanation](https://mlsanity.com/paper/2506.05035/) (2025)

## Built on by (most cited first, 2 of 2)

- [HAROOD: A Benchmark for Out-of-distribution Generalization in Sensor-based Human Activity Recognition](https://mlsanity.com/paper/2512.10807/) (2025)
- [Delta-XAI: A Unified Framework for Explaining Prediction Changes in Online Time Series Monitoring](https://mlsanity.com/paper/2511.23036/) (2025)

## Abstract (arXiv)

This study proposes DeltaSHAP, a novel explainable artificial intelligence (XAI) algorithm specifically designed for online patient monitoring systems. In clinical environments, discovering the causes driving patient risk evolution is critical for timely intervention, yet existing XAI methods fail to address the unique requirements of clinical time series explanation tasks. To this end, DeltaSHAP addresses three key clinical needs: explaining the changes in the consecutive predictions rather than isolated prediction scores, providing both magnitude and direction of feature attributions, and delivering these insights in real time. By adapting Shapley values to temporal settings, our approach accurately captures feature coalition effects. It further attributes prediction changes using only the actually observed feature combinations, making it efficient and practical for time-sensitive clinical applications. We also introduce new evaluation metrics to evaluate the faithfulness of the attributions for online time series, and demonstrate through experiments on online patient monitoring tasks that DeltaSHAP outperforms state-of-the-art XAI methods in both explanation quality as 62% and computational efficiency as 33% time reduction on the MIMIC-III decompensation benchmark. We release our code at https://github.com/AITRICS/DeltaSHAP.