# Feature Importance for Time Series Data: Improving KernelSHAP

arXiv: https://arxiv.org/abs/2210.02176 · HTML: https://mlsanity.com/paper/2210.02176/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Model-Agnostic Counterfactual Explanations for Consequential Decisions](https://mlsanity.com/paper/1905.11190/) (2019)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [FastSHAP: Real-Time Shapley Value Estimation](https://mlsanity.com/paper/2107.07436/) (2021)
- [Explaining Recurrent Neural Network Predictions in Sentiment Analysis](https://mlsanity.com/paper/1706.07206/) (2017)
- [TimeSHAP: Explaining Recurrent Models through Sequence Perturbations](https://mlsanity.com/paper/2012.00073/) (2020)
- [Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs](https://mlsanity.com/paper/1801.05453/) (2018)
- [Interpretability in deep learning for finance: a case study for the Heston model](https://mlsanity.com/paper/2104.09476/) (2021)

## Built on by (most cited first, 5 of 5)

- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [TimeInf: Time Series Data Contribution via Influence Functions](https://mlsanity.com/paper/2407.15247/) (2024)
- [Explaining Clustering of Ecological Momentary Assessment Data Through Temporal and Feature Attention](https://mlsanity.com/paper/2405.04854/) (2024)
- [On the Connection between Game-Theoretic Feature Attributions and Counterfactual Explanations](https://mlsanity.com/paper/2307.06941/) (2023)
- [Autorelevance function and other feature relevance measures for univariate time series](https://mlsanity.com/paper/2607.01959/) (2026)

## Abstract (arXiv)

Feature importance techniques have enjoyed widespread attention in the explainable AI literature as a means of determining how trained machine learning models make their predictions. We consider Shapley value based approaches to feature importance, applied in the context of time series data. We present closed form solutions for the SHAP values of a number of time series models, including VARMAX. We also show how KernelSHAP can be applied to time series tasks, and how the feature importances that come from this technique can be combined to perform "event detection". Finally, we explore the use of Time Consistent Shapley values for feature importance.