# Time Series Model Attribution Visualizations as Explanations

arXiv: https://arxiv.org/abs/2109.12935 · HTML: https://mlsanity.com/paper/2109.12935/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning](https://mlsanity.com/paper/1908.00087/) (2019)
- [Towards a Rigorous Evaluation of XAI Methods on Time Series](https://mlsanity.com/paper/1909.07082/) (2019)
- [Evaluating Recurrent Neural Network Explanations](https://mlsanity.com/paper/1904.11829/) (2019)
- [TS-MULE: Local Interpretable Model-Agnostic Explanations for Time Series Forecast Models](https://mlsanity.com/paper/2109.08438/) (2021)
- [TSViz: Demystification of Deep Learning Models for Time-Series Analysis](https://mlsanity.com/paper/1802.02952/) (2018)
- [Deep neural networks can predict mortality from 12-lead electrocardiogram voltage data](https://mlsanity.com/paper/1904.07032/) (2019)
- [A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems](https://mlsanity.com/paper/1811.11839/) (2018)
- [timeXplain -- A Framework for Explaining the Predictions of Time Series Classifiers](https://mlsanity.com/paper/2007.07606/) (2020)
- [TSInsight: A local-global attribution framework for interpretability in time-series data](https://mlsanity.com/paper/2004.02958/) (2020)
- [An Empirical Study of Explainable AI Techniques on Deep Learning Models For Time Series Tasks](https://mlsanity.com/paper/2012.04344/) (2020)

## Built on by (most cited first, 8 of 8)

- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [A Deep Dive into Perturbations as Evaluation Technique for Time Series XAI](https://mlsanity.com/paper/2307.05104/) (2023)
- [Visual Explanations with Attributions and Counterfactuals on Time Series Classification](https://mlsanity.com/paper/2307.08494/) (2023)
- [Revisiting Backdoor Attacks on Time Series Classification in the Frequency Domain](https://mlsanity.com/paper/2503.09712/) (2025)
- [Introducing the Attribution Stability Indicator: a Measure for Time Series XAI Attributions](https://mlsanity.com/paper/2310.04178/) (2023)
- [Interactive dense pixel visualizations for time series and model attribution explanations](https://mlsanity.com/paper/2408.15073/) (2024)
- [FREQuency ATTribution: benchmarking frequency-based occlusion for time series data](https://mlsanity.com/paper/2506.18481/) (2025)
- [L2GTX: From Local to Global Time Series Explanations](https://mlsanity.com/paper/2603.13065/) (2026)

## Abstract (arXiv)

Attributions are a common local explanation technique for deep learning models on single samples as they are easily extractable and demonstrate the relevance of input values. In many cases, heatmaps visualize such attributions for samples, for instance, on images. However, heatmaps are not always the ideal visualization to explain certain model decisions for other data types. In this review, we focus on attribution visualizations for time series. We collect attribution heatmap visualizations and some alternatives, discuss the advantages as well as disadvantages and give a short position towards future opportunities for attributions and explanations for time series.