# Visual Explanations with Attributions and Counterfactuals on Time Series Classification

arXiv: https://arxiv.org/abs/2307.08494 · HTML: https://mlsanity.com/paper/2307.08494/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 71 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018)
- [LSTM Fully Convolutional Networks for Time Series Classification](https://mlsanity.com/paper/1709.05206/) (2017)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations](https://mlsanity.com/paper/1904.02323/) (2019)
- [explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning](https://mlsanity.com/paper/1908.00087/) (2019)
- [Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models](https://mlsanity.com/paper/1804.09299/) (2018)
- [Towards a Rigorous Evaluation of XAI Methods on Time Series](https://mlsanity.com/paper/1909.07082/) (2019)
- [Instance-based Counterfactual Explanations for Time Series Classification](https://mlsanity.com/paper/2009.13211/) (2020)
- [RuleMatrix: Visualizing and Understanding Classifiers with Rules](https://mlsanity.com/paper/1807.06228/) (2018)
- [Towards Better Analysis of Machine Learning Models: A Visual Analytics Perspective](https://mlsanity.com/paper/1702.01226/) (2017)
- [An Empirical Evaluation of Deep Learning on Highway Driving](https://mlsanity.com/paper/1504.01716/) (2015)
- [TSViz: Demystification of Deep Learning Models for Time-Series Analysis](https://mlsanity.com/paper/1802.02952/) (2018)
- [A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems](https://mlsanity.com/paper/1811.11839/) (2018)

## Built on by (most cited first, 5 of 5)

- [What-If Explanations Over Time: Counterfactuals for Time Series Classification](https://mlsanity.com/paper/2603.27792/) (2026)
- [Interactive Counterfactual Generation for Univariate Time Series](https://mlsanity.com/paper/2408.10633/) (2024)
- [Towards Explainable Deep Clustering for Time Series Data](https://mlsanity.com/paper/2507.20840/) (2025)
- [Finding the DeepDream for Time Series: Activation Maximization for Univariate Time Series](https://mlsanity.com/paper/2408.10628/) (2024)
- [IMFACT: Counterfactual Explanations for Time Series via Intrinsic Mode Function Substitution](https://mlsanity.com/paper/2608.04777/) (2026)

## Abstract (arXiv)

With the rising necessity of explainable artificial intelligence (XAI), we see an increase in task-dependent XAI methods on varying abstraction levels. XAI techniques on a global level explain model behavior and on a local level explain sample predictions. We propose a visual analytics workflow to support seamless transitions between global and local explanations, focusing on attributions and counterfactuals on time series classification. In particular, we adapt local XAI techniques (attributions) that are developed for traditional datasets (images, text) to analyze time series classification, a data type that is typically less intelligible to humans. To generate a global overview, we apply local attribution methods to the data, creating explanations for the whole dataset. These explanations are projected onto two dimensions, depicting model behavior trends, strategies, and decision boundaries. To further inspect the model decision-making as well as potential data errors, a what-if analysis facilitates hypothesis generation and verification on both the global and local levels. We constantly collected and incorporated expert user feedback, as well as insights based on their domain knowledge, resulting in a tailored analysis workflow and system that tightly integrates time series transformations into explanations. Lastly, we present three use cases, verifying that our technique enables users to (1)~explore data transformations and feature relevance, (2)~identify model behavior and decision boundaries, as well as, (3)~the reason for misclassifications.