# Explainable AI for Time Series via Virtual Inspection Layers

arXiv: https://arxiv.org/abs/2303.06365 · HTML: https://mlsanity.com/paper/2303.06365/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 55 references that are in the map.
- First posted on arXiv 11 Mar 2023.

## Builds on (most cited first, 29 of 29)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation](https://mlsanity.com/paper/2206.03208/) (2022)
- [Deep Learning for Audio Signal Processing](https://mlsanity.com/paper/1905.00078/) (2019)
- [ECG Heartbeat Classification: A Deep Transferable Representation](https://mlsanity.com/paper/1805.00794/) (2018)
- [AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmark](https://mlsanity.com/paper/1807.03418/) (2018)
- [Towards Best Practice in Explaining Neural Network Decisions with LRP](https://mlsanity.com/paper/1910.09840/) (2019)
- [Towards a Rigorous Evaluation of XAI Methods on Time Series](https://mlsanity.com/paper/1909.07082/) (2019)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees](https://mlsanity.com/paper/2301.11911/) (2023)
- [Counterfactual Explanations for Machine Learning on Multivariate Time Series Data](https://mlsanity.com/paper/2008.10781/) (2020)
- [This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation](https://mlsanity.com/paper/2108.12204/) (2021)
- [Interpretable Transformations with Encoder-Decoder Networks](https://mlsanity.com/paper/1710.07307/) (2017)
- [TSViz: Demystification of Deep Learning Models for Time-Series Analysis](https://mlsanity.com/paper/1802.02952/) (2018)
- [Interpretable Deep Neural Networks for Single-Trial EEG Classification](https://mlsanity.com/paper/1604.08201/) (2016)
- [Detecting and interpreting myocardial infarction using fully convolutional neural networks](https://mlsanity.com/paper/1806.07385/) (2018)
- [On the Explanation of Machine Learning Predictions in Clinical Gait Analysis](https://mlsanity.com/paper/1912.07737/) (2019)
- [Concept-based model explanations for Electronic Health Records](https://mlsanity.com/paper/2012.02308/) (2020)
- [NeuralHydrology -- Interpreting LSTMs in Hydrology](https://mlsanity.com/paper/1903.07903/) (2019)
- [Trading via Image Classification](https://mlsanity.com/paper/1907.10046/) (2019)
- [Interpreting Deep Neural Networks for Single-Lead ECG Arrhythmia Classification](https://mlsanity.com/paper/2004.05399/) (2020)
- [Comparing Time and Frequency Domain for Audio Event Recognition Using Deep Learning](https://mlsanity.com/paper/1603.05824/) (2016)

## Built on by (most cited first, 23 of 23)

- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [The Clever Hans Effect in Unsupervised Learning](https://mlsanity.com/paper/2408.08041/) (2024)
- [Explaining Predictive Uncertainty by Exposing Second-Order Effects](https://mlsanity.com/paper/2401.17441/) (2024)
- [Right on Time: Revising Time Series Models by Constraining their Explanations](https://mlsanity.com/paper/2402.12921/) (2024)
- [Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Features](https://mlsanity.com/paper/2408.17198/) (2024)
- [FLEXtime: Filterbank learning to explain time series](https://mlsanity.com/paper/2411.05841/) (2024)
- [FreqRISE: Explaining time series using frequency masking](https://mlsanity.com/paper/2406.13584/) (2024)
- [XAI-based Comparison of Input Representations for Audio Event Classification](https://mlsanity.com/paper/2304.14019/) (2023)
- [Explanation Space: A New Perspective into Time Series Interpretability](https://mlsanity.com/paper/2409.01354/) (2024)
- [Multi-SpaCE: Multi-Objective Subsequence-based Sparse Counterfactual Explanations for Multivariate Time Series Classification](https://mlsanity.com/paper/2501.04009/) (2024)
- [Audio Explanation Synthesis with Generative Foundation Models](https://mlsanity.com/paper/2410.07530/) (2024)
- [Time series saliency maps: explaining models across multiple domains](https://mlsanity.com/paper/2505.13100/) (2025)
- [Towards Explainable Deep Clustering for Time Series Data](https://mlsanity.com/paper/2507.20840/) (2025)
- [XpertAI: uncovering regression model strategies for sub-manifolds](https://mlsanity.com/paper/2403.07486/) (2024)
- [On the Necessity of Multi-Domain Explanation: An Uncertainty Principle Approach for Deep Time Series Models](https://mlsanity.com/paper/2506.03267/) (2025)
- [Real-Time Human Activity Recognition on Edge Microcontrollers: Dynamic Hierarchical Inference with Multi-Spectral Sensor Fusion](https://mlsanity.com/paper/2602.00152/) (2026)
- [Explainable AI Using Inherently Interpretable Components for Wearable-based Health Monitoring](https://mlsanity.com/paper/2603.12880/) (2026)
- [L2GTX: From Local to Global Time Series Explanations](https://mlsanity.com/paper/2603.13065/) (2026)
- [CENDRe: Concept Extraction with Natural Domain Representations](https://mlsanity.com/paper/2607.29621/) (2026)
- [EEG-PRISM: Physiologically-Grounded Interpretability of Predictions by EEG Foundation Models](https://mlsanity.com/paper/2608.13676/) (2026)
- [HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives](https://mlsanity.com/paper/2609.01282/) (2026)
- [Learnable Time-Frequency Masks for Explaining Time-Series Classifiers](https://mlsanity.com/paper/2609.29270/) (2026)
- [Explaining Hyperbolic Neural Networks via Geometry-Aware Relevance Propagation](https://mlsanity.com/paper/2609.35128/) (2026)

## Abstract (arXiv)

The field of eXplainable Artificial Intelligence (XAI) has greatly advanced in recent years, but progress has mainly been made in computer vision and natural language processing. For time series, where the input is often not interpretable, only limited research on XAI is available. In this work, we put forward a virtual inspection layer, that transforms the time series to an interpretable representation and allows to propagate relevance attributions to this representation via local XAI methods like layer-wise relevance propagation (LRP). In this way, we extend the applicability of a family of XAI methods to domains (e.g. speech) where the input is only interpretable after a transformation. Here, we focus on the Fourier transformation which is prominently applied in the interpretation of time series and LRP and refer to our method as DFT-LRP. We demonstrate the usefulness of DFT-LRP in various time series classification settings like audio and electronic health records. We showcase how DFT-LRP reveals differences in the classification strategies of models trained in different domains (e.g., time vs. frequency domain) or helps to discover how models act on spurious correlations in the data.