# Benchmarking Deep Learning Interpretability in Time Series Predictions

arXiv: https://arxiv.org/abs/2010.13924 · HTML: https://mlsanity.com/paper/2010.13924/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2020.

## Built on by (most cited first, 30 of 102)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Salient ImageNet: How to discover spurious features in Deep Learning?](https://mlsanity.com/paper/2110.04301/) (2021)
- [HIVE: Evaluating the Human Interpretability of Visual Explanations](https://mlsanity.com/paper/2112.03184/) (2021)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement](https://mlsanity.com/paper/2301.03028/) (2023)
- [Chaos as an interpretable benchmark for forecasting and data-driven modelling](https://mlsanity.com/paper/2110.05266/) (2021)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Improving Deep Learning Interpretability by Saliency Guided Training](https://mlsanity.com/paper/2111.14338/) (2021)
- [Evaluation of post-hoc interpretability methods in time-series classification](https://mlsanity.com/paper/2202.05656/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [TimeSeriesExam: A time series understanding exam](https://mlsanity.com/paper/2410.14752/) (2024)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop](https://mlsanity.com/paper/2503.01013/) (2025)
- [Temporal Dependencies in Feature Importance for Time Series Predictions](https://mlsanity.com/paper/2107.14317/) (2021)
- [Inherently Interpretable Time Series Classification via Multiple Instance Learning](https://mlsanity.com/paper/2311.10049/) (2023)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [PredDiff: Explanations and Interactions from Conditional Expectations](https://mlsanity.com/paper/2102.13519/) (2021)
- [Interpretable Mixture of Experts](https://mlsanity.com/paper/2206.02107/) (2022)
- [Out-of-Distribution Generalization in Time Series: A Survey](https://mlsanity.com/paper/2503.13868/) (2025)
- [MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset](https://mlsanity.com/paper/2102.06761/) (2021)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [Core Risk Minimization using Salient ImageNet](https://mlsanity.com/paper/2203.15566/) (2022)
- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [Deletion and Insertion Tests in Regression Models](https://mlsanity.com/paper/2205.12423/) (2022)
- [Generating Sparse Counterfactual Explanations For Multivariate Time Series](https://mlsanity.com/paper/2206.00931/) (2022)
- [dCAM: Dimension-wise Class Activation Map for Explaining Multivariate Data Series Classification](https://mlsanity.com/paper/2207.12165/) (2022)
- [Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices](https://mlsanity.com/paper/2203.14177/) (2022)
- [Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition](https://mlsanity.com/paper/2305.18011/) (2023)
- [TIMING: Temporality-Aware Integrated Gradients for Time Series Explanation](https://mlsanity.com/paper/2506.05035/) (2025)

## Abstract (arXiv)

Saliency methods are used extensively to highlight the importance of input features in model predictions. These methods are mostly used in vision and language tasks, and their applications to time series data is relatively unexplored. In this paper, we set out to extensively compare the performance of various saliency-based interpretability methods across diverse neural architectures, including Recurrent Neural Network, Temporal Convolutional Networks, and Transformers in a new benchmark of synthetic time series data. We propose and report multiple metrics to empirically evaluate the performance of saliency methods for detecting feature importance over time using both precision (i.e., whether identified features contain meaningful signals) and recall (i.e., the number of features with signal identified as important). Through several experiments, we show that (i) in general, network architectures and saliency methods fail to reliably and accurately identify feature importance over time in time series data, (ii) this failure is mainly due to the conflation of time and feature domains, and (iii) the quality of saliency maps can be improved substantially by using our proposed two-step temporal saliency rescaling (TSR) approach that first calculates the importance of each time step before calculating the importance of each feature at a time step.