# Temporal Dependencies in Feature Importance for Time Series Predictions

arXiv: https://arxiv.org/abs/2107.14317 · HTML: https://mlsanity.com/paper/2107.14317/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 22 of 22)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Explaining by Removing: A Unified Framework for Model Explanation](https://mlsanity.com/paper/2011.14878/) (2020)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [Attend and Diagnose: Clinical Time Series Analysis using Attention Models](https://mlsanity.com/paper/1711.03905/) (2017)
- [Benchmarking Deep Learning Interpretability in Time Series Predictions](https://mlsanity.com/paper/2010.13924/) (2020)
- [Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study](https://mlsanity.com/paper/2002.00772/) (2020)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data](https://mlsanity.com/paper/1807.08820/) (2018)
- [Clinical Intervention Prediction and Understanding using Deep Networks](https://mlsanity.com/paper/1705.08498/) (2017)
- [Improving performance of deep learning models with axiomatic attribution priors and expected gradients](https://mlsanity.com/paper/1906.10670/) (2019)
- [A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems](https://mlsanity.com/paper/1811.11839/) (2018)

## Built on by (most cited first, 27 of 27)

- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop](https://mlsanity.com/paper/2503.01013/) (2025)
- [TimeX++: Learning Time-Series Explanations with Information Bottleneck](https://mlsanity.com/paper/2405.09308/) (2024)
- [Inherently Interpretable Time Series Classification via Multiple Instance Learning](https://mlsanity.com/paper/2311.10049/) (2023)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [TIMING: Temporality-Aware Integrated Gradients for Time Series Explanation](https://mlsanity.com/paper/2506.05035/) (2025)
- [Robust Explainer Recommendation for Time Series Classification](https://mlsanity.com/paper/2306.05501/) (2023)
- [PaAno: Patch-Based Representation Learning for Time-Series Anomaly Detection](https://mlsanity.com/paper/2602.01359/) (2026)
- [EDformer: Embedded Decomposition Transformer for Interpretable Multivariate Time Series Predictions](https://mlsanity.com/paper/2412.12227/) (2024)
- [ShapeX: Shapelet-Driven Post Hoc Explanations for Time Series Classification Models](https://mlsanity.com/paper/2510.20084/) (2025)
- [Latent Temporal Flows for Multivariate Analysis of Wearables Data](https://mlsanity.com/paper/2210.07475/) (2022)
- [Time-series attribution maps with regularized contrastive learning](https://mlsanity.com/paper/2502.12977/) (2025)
- [DeltaSHAP: Explaining Prediction Evolutions in Online Patient Monitoring with Shapley Values](https://mlsanity.com/paper/2507.02342/) (2025)
- [GroupSegment-SHAP: Shapley Value Explanations with Group-Segment Players for Multivariate Time Series](https://mlsanity.com/paper/2601.06114/) (2026)
- [EXCODER: EXplainable Classification Of DiscretE time series Representations](https://mlsanity.com/paper/2602.13087/) (2026)
- [The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations](https://mlsanity.com/paper/2607.16236/) (2026)
- [Delta-XAI: A Unified Framework for Explaining Prediction Changes in Online Time Series Monitoring](https://mlsanity.com/paper/2511.23036/) (2025)
- [A Self-explainable Model of Long Time Series by Extracting Informative Structured Causal Patterns](https://mlsanity.com/paper/2512.01412/) (2025)
- [When, How Long and How Much? Interpretable Neural Networks for Time Series Regression by Learning to Mask and Aggregate](https://mlsanity.com/paper/2512.03578/) (2025)
- [TimeSAE: Causal Sparse Decoding for Faithful Explanations of Black-Box Time Series Models](https://mlsanity.com/paper/2601.09776/) (2026)
- [Interpretability in Deep Time Series Models Demands Semantic Alignment](https://mlsanity.com/paper/2602.02239/) (2026)
- [Exposing Vulnerabilities in Explanation for Time Series Classifiers via Dual-Target Attacks](https://mlsanity.com/paper/2602.02763/) (2026)
- [Temporal Functional Circuits: From Spline Plots to Faithful Explanations in KAN Forecasting](https://mlsanity.com/paper/2605.05685/) (2026)
- [Global Explanations for Multivariate Time Series Forecasting Models via K-Order Markov Approximations](https://mlsanity.com/paper/2606.27599/) (2026)
- [Beyond Sufficiency: Time Series Explanation with Counterfactual Necessity](https://mlsanity.com/paper/2607.21573/) (2026)
- [Towards A Unified Information Bottleneck Framework for Time Series Explanations](https://mlsanity.com/paper/2608.25897/) (2026)
- [Explaining Time Series Forecasting with Horizon-Resolved Attribution](https://mlsanity.com/paper/2609.12639/) (2026)

## Abstract (arXiv)

Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time. In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues. Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score. Furthermore, WinIT captures the varying importance of a feature over time, by summarizing its importance over a window of past time steps. We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods, and explore the impact of various evaluation strategies. Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics. The code for our work is publicly available at urlhttps://github.com/layer6ai-labs/WinIT.