mlsanityData to 7 Oct 2026

Temporal Dependencies in Feature Importance for Time Series Predictions

arXiv 2107.14317 ↗First posted 29 Jul 2021cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time. In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues. Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score. Furthermore, WinIT captures the varying importance of a feature over time, by summarizing its importance over a window of past time steps. We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods, and explore the impact of various evaluation strategies. Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics. The code for our work is publicly available at urlhttps://github.com/layer6ai-labs/WinIT.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 29 Jul 2021.

Builds on

22 of 33 references.

References · 22

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  5. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  7. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  8. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  9. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  10. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  11. 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 344
  12. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
10 more
  1. 2018Explaining Image Classifiers by Counterfactual Generationcited by 144
  2. 2019MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIcited by 121
  3. 2017Attend and Diagnose: Clinical Time Series Analysis using Attention Modelscited by 105
  4. 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
  5. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  6. 2021Explaining Time Series Predictions with Dynamic Maskscited by 72
  7. 2018RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Datacited by 47
  8. 2017Clinical Intervention Prediction and Understanding using Deep Networkscited by 35
  9. 2019Improving performance of deep learning models with axiomatic attribution priors and expected gradientscited by 32
  10. 2018A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systemscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 3; 2025: 8; 2026: 12.

27 papers.

Sort:

  1. 2023Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistencycited by 42
  2. 2025TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loopcited by 39
  3. 2024TimeX++: Learning Time-Series Explanations with Information Bottleneckcited by 32
  4. 2023Inherently Interpretable Time Series Classification via Multiple Instance Learningcited by 23
  5. 2024Explaining Time Series via Contrastive and Locally Sparse Perturbationscited by 22
  6. 2025TIMING: Temporality-Aware Integrated Gradients for Time Series Explanationcited by 9
  7. 2023Robust Explainer Recommendation for Time Series Classificationcited by 7
  8. 2024EDformer: Embedded Decomposition Transformer for Interpretable Multivariate Time Series Predictionscited by 5
  9. 2026PaAno: Patch-Based Representation Learning for Time-Series Anomaly Detectioncited by 5
  10. 2025ShapeX: Shapelet-Driven Post Hoc Explanations for Time Series Classification Modelscited by 3
  11. 2022Latent Temporal Flows for Multivariate Analysis of Wearables Datacited by 2
  12. 2025Time-series attribution maps with regularized contrastive learningcited by 2
15 more
  1. 2025DeltaSHAP: Explaining Prediction Evolutions in Online Patient Monitoring with Shapley Valuescited by 2
  2. 2026GroupSegment-SHAP: Shapley Value Explanations with Group-Segment Players for Multivariate Time Seriescited by 1
  3. 2026EXCODER: EXplainable Classification Of DiscretE time series Representationscited by 1
  4. 2026The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanationscited by 1
  5. 2025Delta-XAI: A Unified Framework for Explaining Prediction Changes in Online Time Series Monitoringcited by 0
  6. 2025A Self-explainable Model of Long Time Series by Extracting Informative Structured Causal Patternscited by 0
  7. 2025When, How Long and How Much? Interpretable Neural Networks for Time Series Regression by Learning to Mask and Aggregatecited by 0
  8. 2026TimeSAE: Causal Sparse Decoding for Faithful Explanations of Black-Box Time Series Modelscited by 0
  9. 2026Interpretability in Deep Time Series Models Demands Semantic Alignmentcited by 0
  10. 2026Exposing Vulnerabilities in Explanation for Time Series Classifiers via Dual-Target Attackscited by 0
  11. 2026Temporal Functional Circuits: From Spline Plots to Faithful Explanations in KAN Forecastingcited by 0
  12. 2026Global Explanations for Multivariate Time Series Forecasting Models via K-Order Markov Approximationscited by 0
  13. 2026Beyond Sufficiency: Time Series Explanation with Counterfactual Necessitycited by 0
  14. 2026Towards A Unified Information Bottleneck Framework for Time Series Explanationscited by 0
  15. 2026Explaining Time Series Forecasting with Horizon-Resolved Attributioncited by 0