# Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Survey

arXiv: https://arxiv.org/abs/2104.00950 · HTML: https://mlsanity.com/paper/2104.00950/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 102 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers](https://mlsanity.com/paper/1604.00825/) (2016)
- [LSTM Fully Convolutional Networks for Time Series Classification](https://mlsanity.com/paper/1709.05206/) (2017)
- [Adversarial Robustness on In- and Out-Distribution Improves Explainability](https://mlsanity.com/paper/2003.09461/) (2020)
- [What Does Explainable AI Really Mean? A New Conceptualization of Perspectives](https://mlsanity.com/paper/1710.00794/) (2017)
- [Towards a Rigorous Evaluation of XAI Methods on Time Series](https://mlsanity.com/paper/1909.07082/) (2019)
- [Clinical Intervention Prediction and Understanding using Deep Networks](https://mlsanity.com/paper/1705.08498/) (2017)
- [Interpretable Time Series Classification using Linear Models and Multi-resolution Multi-domain Symbolic Representations](https://mlsanity.com/paper/2006.01667/) (2020)
- [Explaining Deep Classification of Time-Series Data with Learned Prototypes](https://mlsanity.com/paper/1904.08935/) (2019)
- [Fooling a Real Car with Adversarial Traffic Signs](https://mlsanity.com/paper/1907.00374/) (2019)
- [TSViz: Demystification of Deep Learning Models for Time-Series Analysis](https://mlsanity.com/paper/1802.02952/) (2018)
- [Post-hoc explanation of black-box classifiers using confident itemsets](https://mlsanity.com/paper/2005.01992/) (2020)
- [ConvTimeNet: A Pre-trained Deep Convolutional Neural Network for Time Series Classification](https://mlsanity.com/paper/1904.12546/) (2019)
- [Detecting and interpreting myocardial infarction using fully convolutional neural networks](https://mlsanity.com/paper/1806.07385/) (2018)
- [Reliable and Explainable Machine Learning Methods for Accelerated Material Discovery](https://mlsanity.com/paper/1901.02717/) (2019)
- [Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks](https://mlsanity.com/paper/1908.07319/) (2019)
- [Interpreting Deep Classifier by Visual Distillation of Dark Knowledge](https://mlsanity.com/paper/1803.04042/) (2018)
- [Focusing on What is Relevant: Time-Series Learning and Understanding using Attention](https://mlsanity.com/paper/1806.08523/) (2018)

## Built on by (most cited first, 30 of 72)

- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2308.05566/) (2023)
- [Toward Explainable AI for Regression Models](https://mlsanity.com/paper/2112.11407/) (2021)
- [Explainable AI for clinical and remote health applications: a survey on tabular and time series data](https://mlsanity.com/paper/2209.06528/) (2022)
- [WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Values](https://mlsanity.com/paper/2211.06507/) (2022)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [A Comprehensive Review on Financial Explainable AI](https://mlsanity.com/paper/2309.11960/) (2023)
- [Explainable Predictive Maintenance: A Survey of Current Methods, Challenges and Opportunities](https://mlsanity.com/paper/2401.07871/) (2024)
- [Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey](https://mlsanity.com/paper/2206.04800/) (2022)
- [Which LIME should I trust? Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2503.24365/) (2025)
- [Right on Time: Revising Time Series Models by Constraining their Explanations](https://mlsanity.com/paper/2402.12921/) (2024)
- [Vision Paper: Causal Inference for Interpretable and Robust Machine Learning in Mobility Analysis](https://mlsanity.com/paper/2210.10010/) (2022)
- [Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations](https://mlsanity.com/paper/2310.08537/) (2023)
- [AutoXPCR: Automated Multi-Objective Model Selection for Time Series Forecasting](https://mlsanity.com/paper/2312.13038/) (2023)
- [FLEXtime: Filterbank learning to explain time series](https://mlsanity.com/paper/2411.05841/) (2024)
- [TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems](https://mlsanity.com/paper/2604.05364/) (2026)
- [ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniques](https://mlsanity.com/paper/2209.03798/) (2022)
- [On the Robustness of Explanations of Deep Neural Network Models: A Survey](https://mlsanity.com/paper/2211.04780/) (2022)
- [How Human-Centered Explainable AI Interface Are Designed and Evaluated: A Systematic Survey](https://mlsanity.com/paper/2403.14496/) (2024)
- [Don't Get Me Wrong: How to Apply Deep Visual Interpretations to Time Series](https://mlsanity.com/paper/2203.07861/) (2022)
- [TSInterpret: A unified framework for time series interpretability](https://mlsanity.com/paper/2208.05280/) (2022)
- [Painting the black box white: experimental findings from applying XAI to an ECG reading setting](https://mlsanity.com/paper/2210.15236/) (2022)
- [The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research](https://mlsanity.com/paper/2310.07882/) (2023)
- [Explaining Time Series Classification Predictions via Causal Attributions](https://mlsanity.com/paper/2405.15871/) (2024)
- [FreqRISE: Explaining time series using frequency masking](https://mlsanity.com/paper/2406.13584/) (2024)
- [Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review](https://mlsanity.com/paper/2408.14491/) (2024)
- [An Overview of Violence Detection Techniques: Current Challenges and Future Directions](https://mlsanity.com/paper/2209.11680/) (2022)

## Abstract (arXiv)

Most of state of the art methods applied on time series consist of deep learning methods that are too complex to be interpreted. This lack of interpretability is a major drawback, as several applications in the real world are critical tasks, such as the medical field or the autonomous driving field. The explainability of models applied on time series has not gather much attention compared to the computer vision or the natural language processing fields. In this paper, we present an overview of existing explainable AI (XAI) methods applied on time series and illustrate the type of explanations they produce. We also provide a reflection on the impact of these explanation methods to provide confidence and trust in the AI systems.