# TimeSHAP: Explaining Recurrent Models through Sequence Perturbations

arXiv: https://arxiv.org/abs/2012.00073 · HTML: https://mlsanity.com/paper/2012.00073/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 30 of 74)

- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop](https://mlsanity.com/paper/2503.01013/) (2025)
- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [TimeX++: Learning Time-Series Explanations with Information Bottleneck](https://mlsanity.com/paper/2405.09308/) (2024)
- [SurvSHAP(t): Time-dependent explanations of machine learning survival models](https://mlsanity.com/paper/2208.11080/) (2022)
- [WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Values](https://mlsanity.com/paper/2211.06507/) (2022)
- [Inherently Interpretable Time Series Classification via Multiple Instance Learning](https://mlsanity.com/paper/2311.10049/) (2023)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [A Comparative Study of Methods for Estimating Conditional Shapley Values and When to Use Them](https://mlsanity.com/paper/2305.09536/) (2023)
- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing](https://mlsanity.com/paper/2205.04559/) (2022)
- [Diverse Counterfactual Explanations for Anomaly Detection in Time Series](https://mlsanity.com/paper/2203.11103/) (2022)
- [Ensembles of Random SHAPs](https://mlsanity.com/paper/2103.03302/) (2021)
- [Feature Importance Explanations for Temporal Black-Box Models](https://mlsanity.com/paper/2102.11934/) (2021)
- [An Imprecise SHAP as a Tool for Explaining the Class Probability Distributions under Limited Training Data](https://mlsanity.com/paper/2106.09111/) (2021)
- [Class-Specific Explainability for Deep Time Series Classifiers](https://mlsanity.com/paper/2210.05411/) (2022)
- [TimeInf: Time Series Data Contribution via Influence Functions](https://mlsanity.com/paper/2407.15247/) (2024)
- [Feature Importance for Time Series Data: Improving KernelSHAP](https://mlsanity.com/paper/2210.02176/) (2022)
- [TemporAI: Facilitating Machine Learning Innovation in Time Domain Tasks for Medicine](https://mlsanity.com/paper/2301.12260/) (2023)
- [Unifying Prediction and Explanation in Time-Series Transformers via Shapley-based Pretraining](https://mlsanity.com/paper/2501.15070/) (2025)
- [GlyTwin: Digital Twin for Glucose Control in Type 1 Diabetes Through Optimal Behavioral Modifications Using Patient-Centric Counterfactuals](https://mlsanity.com/paper/2504.09846/) (2025)
- [Step-resolved data attribution for looped transformers](https://mlsanity.com/paper/2602.10097/) (2026)
- [CAFO: Feature-Centric Explanation on Time Series Classification](https://mlsanity.com/paper/2406.01833/) (2024)
- [AXIS: Explainable Time Series Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2509.24378/) (2025)
- [Explainable Load Forecasting with Covariate-Informed Time Series Foundation Models](https://mlsanity.com/paper/2604.28149/) (2026)
- [Neural Additive Models for Nowcasting](https://mlsanity.com/paper/2205.10020/) (2022)
- [C-SHAP for time series: An approach to high-level temporal explanations](https://mlsanity.com/paper/2504.11159/) (2025)
- [HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning](https://mlsanity.com/paper/2509.24118/) (2025)
- [Interpretable Ensembles of Hyper-Rectangles as Base Models](https://mlsanity.com/paper/2303.08625/) (2023)

## Abstract (arXiv)

Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agnostic recurrent explainer that builds upon KernelSHAP and extends it to the sequential domain. TimeSHAP computes feature-, timestep-, and cell-level attributions. As sequences may be arbitrarily long, we further propose a pruning method that is shown to dramatically decrease both its computational cost and the variance of its attributions. We use TimeSHAP to explain the predictions of a real-world bank account takeover fraud detection RNN model, and draw key insights from its explanations: i) the model identifies important features and events aligned with what fraud analysts consider cues for account takeover; ii) positive predicted sequences can be pruned to only 10% of the original length, as older events have residual attribution values; iii) the most recent input event of positive predictions only contributes on average to 41% of the model's score; iv) notably high attribution to client's age, suggesting a potential discriminatory reasoning, later confirmed as higher false positive rates for older clients.