# Spatial-temporal Concept based Explanation of 3D ConvNets

arXiv: https://arxiv.org/abs/2206.05275 · HTML: https://mlsanity.com/paper/2206.05275/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Visualizing Deep Neural Network Decisions: Prediction Difference Analysis](https://mlsanity.com/paper/1702.04595/) (2017)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1908.02686/) (2019)
- [A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts](https://mlsanity.com/paper/2105.00290/) (2021)
- [Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions](https://mlsanity.com/paper/1902.01078/) (2019)
- [Hierarchical Attention Network for Action Recognition in Videos](https://mlsanity.com/paper/1607.06416/) (2016)
- [Interpretable Neural Networks with Frank-Wolfe: Sparse Relevance Maps and Relevance Orderings](https://mlsanity.com/paper/2110.08105/) (2021)

## Built on by (most cited first, 12 of 12)

- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding](https://mlsanity.com/paper/2309.12482/) (2023)
- [Natural Language Reinforcement Learning](https://mlsanity.com/paper/2402.07157/) (2024)
- [Natural Language Reinforcement Learning](https://mlsanity.com/paper/2411.14251/) (2024)
- [On The Variability of Concept Activation Vectors](https://mlsanity.com/paper/2509.24058/) (2025)
- [Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition](https://mlsanity.com/paper/2511.03725/) (2025)
- [PCBEAR: Pose Concept Bottleneck for Explainable Action Recognition](https://mlsanity.com/paper/2504.13140/) (2025)
- [LD-ViCE: Latent Diffusion Model for Video Counterfactual Explanations](https://mlsanity.com/paper/2509.08422/) (2025)
- [Concepts in Motion: Temporal Concept Bottleneck Model for Interpretable Video Classification](https://mlsanity.com/paper/2509.20899/) (2025)
- [TraNCE: Transformative Non-linear Concept Explainer for CNNs](https://mlsanity.com/paper/2503.20230/) (2025)
- [ConceptFlow: Hierarchical and Fine-grained Concept-Based Explanation for Convolutional Neural Networks](https://mlsanity.com/paper/2509.18147/) (2025)

## Abstract (arXiv)

Recent studies have achieved outstanding success in explaining 2D image recognition ConvNets. On the other hand, due to the computation cost and complexity of video data, the explanation of 3D video recognition ConvNets is relatively less studied. In this paper, we present a 3D ACE (Automatic Concept-based Explanation) framework for interpreting 3D ConvNets. In our approach: (1) videos are represented using high-level supervoxels, which is straightforward for human to understand; and (2) the interpreting framework estimates a score for each voxel, which reflects its importance in the decision procedure. Experiments show that our method can discover spatial-temporal concepts of different importance-levels, and thus can explore the influence of the concepts on a target task, such as action classification, in-depth. The codes are publicly available.