mlsanityData to 7 Oct 2026

Feature Importance Explanations for Temporal Black-Box Models

arXiv 2102.11934 ↗First posted 23 Feb 2021cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Models in the supervised learning framework may capture rich and complex representations over the features that are hard for humans to interpret. Existing methods to explain such models are often specific to architectures and data where the features do not have a time-varying component. In this work, we propose TIME, a method to explain models that are inherently temporal in nature. Our approach (i) uses a model-agnostic permutation-based approach to analyze global feature importance, (ii) identifies the importance of salient features with respect to their temporal ordering as well as localized windows of influence, and (iii) uses hypothesis testing to provide statistical rigor.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 23 Feb 2021.

Builds on

22 of 46 references.

References · 22

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  5. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 364
  6. 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 344
  7. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  8. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
  9. 2020Problems with Shapley-value-based explanations as feature importance measurescited by 198
  10. 2015Visualizing and Understanding Recurrent Networkscited by 158
  11. 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
  12. 2019CXPlain: Causal Explanations for Model Interpretation under Uncertaintycited by 98
10 more
  1. 2020TimeSHAP: Explaining Recurrent Models through Sequence Perturbationscited by 73
  2. 2017Beyond Sparsity: Tree Regularization of Deep Models for Interpretabilitycited by 68
  3. 2013Correlation and variable importance in random forestscited by 41
  4. 2017Clinical Intervention Prediction and Understanding using Deep Networkscited by 35
  5. 2019Global Explanations of Neural Networks: Mapping the Landscape of Predictionscited by 32
  6. 2019Unbiased Measurement of Feature Importance in Tree-Based Methodscited by 30
  7. 2019Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networkscited by 20
  8. 2014Grouped variable importance with random forests and application to multiple functional data analysiscited by 16
  9. 2018Understanding Learned Models by Identifying Important Features at the Right Resolutioncited by 2
  10. 2018Game-Theoretic Interpretability for Temporal Modelingcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 4.

7 papers.

  1. 2022WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Valuescited by 24
  2. 2023Designing User-Centric Behavioral Interventions to Prevent Dysglycemia with Novel Counterfactual Explanationscited by 5
  3. 2025GlyTwin: Digital Twin for Glucose Control in Type 1 Diabetes Through Optimal Behavioral Modifications Using Patient-Centric Counterfactualscited by 5
  4. 2025HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learningcited by 3
  5. 2023Characterizing climate pathways using feature importance on echo state networkscited by 2
  6. 2025Explainable Time Series Prediction of Tyre Energy in Formula One Race Strategycited by 0
  7. 2025OrdShap: Feature Position Importance for Sequential Black-Box Modelscited by 0