# What went wrong and when? Instance-wise Feature Importance for Time-series Models

arXiv: https://arxiv.org/abs/2003.02821 · HTML: https://mlsanity.com/paper/2003.02821/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2020.

## Built on by (most cited first, 6 of 6)

- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Improving Deep Learning Interpretability by Saliency Guided Training](https://mlsanity.com/paper/2111.14338/) (2021)
- [Clairvoyance: A Pipeline Toolkit for Medical Time Series](https://mlsanity.com/paper/2310.18688/) (2023)
- [Class-Specific Explainability for Deep Time Series Classifiers](https://mlsanity.com/paper/2210.05411/) (2022)
- [Explainable AI for tailored electricity consumption feedback -- an experimental evaluation of visualizations](https://mlsanity.com/paper/2208.11408/) (2022)
- [Counterfactual Explanations and Predictive Models to Enhance Clinical Decision-Making in Schizophrenia using Digital Phenotyping](https://mlsanity.com/paper/2306.03980/) (2023)

## Abstract (arXiv)

Explanations of time series models are useful for high stakes applications like healthcare but have received little attention in machine learning literature. We propose FIT, a framework that evaluates the importance of observations for a multivariate time-series black-box model by quantifying the shift in the predictive distribution over time. FIT defines the importance of an observation based on its contribution to the distributional shift under a KL-divergence that contrasts the predictive distribution against a counterfactual where the rest of the features are unobserved. We also demonstrate the need to control for time-dependent distribution shifts. We compare with state-of-the-art baselines on simulated and real-world clinical data and demonstrate that our approach is superior in identifying important time points and observations throughout the time series.