# Interpretation of Deep Temporal Representations by Selective Visualization of Internally Activated Nodes

arXiv: https://arxiv.org/abs/2004.12538 · HTML: https://mlsanity.com/paper/2004.12538/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Survey](https://mlsanity.com/paper/2104.00950/) (2021)
- [Explainable AI for clinical and remote health applications: a survey on tabular and time series data](https://mlsanity.com/paper/2209.06528/) (2022)
- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [MTS2Graph: Interpretable Multivariate Time Series Classification with Temporal Evolving Graphs](https://mlsanity.com/paper/2306.03834/) (2023)
- [L2GTX: From Local to Global Time Series Explanations](https://mlsanity.com/paper/2603.13065/) (2026)

## Abstract (arXiv)

Recently deep neural networks demonstrate competitive performances in classification and regression tasks for many temporal or sequential data. However, it is still hard to understand the classification mechanisms of temporal deep neural networks. In this paper, we propose two new frameworks to visualize temporal representations learned from deep neural networks. Given input data and output, our algorithm interprets the decision of temporal neural network by extracting highly activated periods and visualizes a sub-sequence of input data which contributes to activate the units. Furthermore, we characterize such sub-sequences with clustering and calculate the uncertainty of the suggested type and actual data. We also suggest Layer-wise Relevance from the output of a unit, not from the final output, with backward Monte-Carlo dropout to show the relevance scores of each input point to activate units with providing a visual representation of the uncertainty about this impact.