# Discriminating Spatial and Temporal Relevance in Deep Taylor Decompositions for Explainable Activity Recognition

arXiv: https://arxiv.org/abs/1908.01536 · HTML: https://mlsanity.com/paper/1908.01536/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2019.

## Built on by (most cited first, 5 of 5)

- [STAA: Spatio-Temporal Attention Attribution for Real-Time Interpreting Transformer-based Video Models](https://mlsanity.com/paper/2411.00630/) (2024)
- [Temporal Relevance Analysis for Video Action Models](https://mlsanity.com/paper/2204.11929/) (2022)
- [Taylor saves for later: disentanglement for video prediction using Taylor representation](https://mlsanity.com/paper/2105.11062/) (2021)
- [Object-ABN: Learning to Generate Sharp Attention Maps for Action Recognition](https://mlsanity.com/paper/2207.13306/) (2022)
- [REVEX: A Unified Framework for Removal-Based Explainable Artificial Intelligence in Video](https://mlsanity.com/paper/2401.11796/) (2024)

## Abstract (arXiv)

Current techniques for explainable AI have been applied with some success to image processing. The recent rise of research in video processing has called for similar work n deconstructing and explaining spatio-temporal models. While many techniques are designed for 2D convolutional models, others are inherently applicable to any input domain. One such body of work, deep Taylor decomposition, propagates relevance from the model output distributively onto its input and thus is not restricted to image processing models. However, by exploiting a simple technique that removes motion information, we show that it is not the case that this technique is effective as-is for representing relevance in non-image tasks. We instead propose a discriminative method that produces a na"ive representation of both the spatial and temporal relevance of a frame as two separate objects. This new discriminative relevance model exposes relevance in the frame attributed to motion, that was previously ambiguous in the original explanation. We observe the effectiveness of this technique on a range of samples from the UCF-101 action recognition dataset, two of which are demonstrated in this paper.