# WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Values

arXiv: https://arxiv.org/abs/2211.06507 · HTML: https://mlsanity.com/paper/2211.06507/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Attend and Diagnose: Clinical Time Series Analysis using Attention Models](https://mlsanity.com/paper/1711.03905/) (2017)
- [TimeSHAP: Explaining Recurrent Models through Sequence Perturbations](https://mlsanity.com/paper/2012.00073/) (2020)
- [Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Survey](https://mlsanity.com/paper/2104.00950/) (2021)
- [Towards a Rigorous Evaluation of XAI Methods on Time Series](https://mlsanity.com/paper/1909.07082/) (2019)
- [Exploring Interpretable LSTM Neural Networks over Multi-Variable Data](https://mlsanity.com/paper/1905.12034/) (2019)
- [Explainable artificial intelligence model to predict acute critical illness from electronic health records](https://mlsanity.com/paper/1912.01266/) (2019)
- [groupShapley: Efficient prediction explanation with Shapley values for feature groups](https://mlsanity.com/paper/2106.12228/) (2021)
- [Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networks](https://mlsanity.com/paper/1910.12370/) (2019)
- [Ensembles of Random SHAPs](https://mlsanity.com/paper/2103.03302/) (2021)
- [Feature Importance Explanations for Temporal Black-Box Models](https://mlsanity.com/paper/2102.11934/) (2021)
- [An Empirical Comparison of Explainable Artificial Intelligence Methods for Clinical Data: A Case Study on Traumatic Brain Injury](https://mlsanity.com/paper/2208.06717/) (2022)

## Built on by (most cited first, 24 of 24)

- [Inherently Interpretable Time Series Classification via Multiple Instance Learning](https://mlsanity.com/paper/2311.10049/) (2023)
- [FLEXtime: Filterbank learning to explain time series](https://mlsanity.com/paper/2411.05841/) (2024)
- [Explainable Load Forecasting with Covariate-Informed Time Series Foundation Models](https://mlsanity.com/paper/2604.28149/) (2026)
- [C-SHAP for time series: An approach to high-level temporal explanations](https://mlsanity.com/paper/2504.11159/) (2025)
- [HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning](https://mlsanity.com/paper/2509.24118/) (2025)
- [From Prototypes to Sparse ECG Explanations: SHAP-Driven Counterfactuals for Multivariate Time-Series Multi-class Classification](https://mlsanity.com/paper/2510.19514/) (2025)
- [ShapeX: Shapelet-Driven Post Hoc Explanations for Time Series Classification Models](https://mlsanity.com/paper/2510.20084/) (2025)
- [An Unsupervised Deep Explainable AI Framework for Localization of Concurrent Replay Attacks in Nuclear Reactor Signals](https://mlsanity.com/paper/2508.09162/) (2025)
- [Explainable time-series forecasting with sampling-free SHAP for Transformers](https://mlsanity.com/paper/2512.20514/) (2025)
- [Surrogate Modeling for Explainable Predictive Time Series Corrections](https://mlsanity.com/paper/2412.19897/) (2024)
- [Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgment](https://mlsanity.com/paper/2506.01290/) (2025)
- [GroupSegment-SHAP: Shapley Value Explanations with Group-Segment Players for Multivariate Time Series](https://mlsanity.com/paper/2601.06114/) (2026)
- [Explainable Unsupervised Multi-Anomaly Detection and Temporal Localization in Nuclear Times Series Data with a Dual Attention-Based Autoencoder](https://mlsanity.com/paper/2509.12372/) (2025)
- [Lightweight Time Series Data Valuation on Time Series Foundation Models via In-Context Finetuning](https://mlsanity.com/paper/2511.11648/) (2025)
- [Explainable AI to Improve Machine Learning Reliability for Industrial Cyber-Physical Systems](https://mlsanity.com/paper/2601.16074/) (2026)
- [Interpretability in Deep Time Series Models Demands Semantic Alignment](https://mlsanity.com/paper/2602.02239/) (2026)
- [ConTex: Reformulating Counterfactual Generation For Time Series Forecasting](https://mlsanity.com/paper/2606.18049/) (2026)
- [Autorelevance function and other feature relevance measures for univariate time series](https://mlsanity.com/paper/2607.01959/) (2026)
- [Feature Attribution-Based Explainability Analysis of Deep Learning Models in Predictive Process Monitoring](https://mlsanity.com/paper/2607.17783/) (2026)
- [DynSHAP: Towards Explainable Dynamic Survival Analysis](https://mlsanity.com/paper/2609.13042/) (2026)
- [IMPACT-VLA: Interaction-aware Multimodal Propagation Attribution via Counterfactual Trajectories for Vision-Language-Action Policies](https://mlsanity.com/paper/2609.15005/) (2026)
- [Learnable Time-Frequency Masks for Explaining Time-Series Classifiers](https://mlsanity.com/paper/2609.29270/) (2026)
- [A Hierarchy of Entropy-Shapley Games for Multivariate Predictive Uncertainty](https://mlsanity.com/paper/2609.35217/) (2026)
- [Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability](https://mlsanity.com/paper/2610.01168/) (2026)

## Abstract (arXiv)

Unpacking and comprehending how black-box machine learning algorithms make decisions has been a persistent challenge for researchers and end-users. Explaining time-series predictive models is useful for clinical applications with high stakes to understand the behavior of prediction models. However, existing approaches to explain such models are frequently unique to data where the features do not have a time-varying component. In this paper, we introduce WindowSHAP, a model-agnostic framework for explaining time-series classifiers using Shapley values. We intend for WindowSHAP to mitigate the computational complexity of calculating Shapley values for long time-series data as well as improve the quality of explanations. WindowSHAP is based on partitioning a sequence into time windows. Under this framework, we present three distinct algorithms of Stationary, Sliding and Dynamic WindowSHAP, each evaluated against baseline approaches, KernelSHAP and TimeSHAP, using perturbation and sequence analyses metrics. We applied our framework to clinical time-series data from both a specialized clinical domain (Traumatic Brain Injury - TBI) as well as a broad clinical domain (critical care medicine). The experimental results demonstrate that, based on the two quantitative metrics, our framework is superior at explaining clinical time-series classifiers, while also reducing the complexity of computations. We show that for time-series data with 120 time steps (hours), merging 10 adjacent time points can reduce the CPU time of WindowSHAP by 80% compared to KernelSHAP. We also show that our Dynamic WindowSHAP algorithm focuses more on the most important time steps and provides more understandable explanations. As a result, WindowSHAP not only accelerates the calculation of Shapley values for time-series data, but also delivers more understandable explanations with higher quality.