Spatio-Temporal Perturbations for Video Attribution
Not on the map. Why?
Abstract (arXiv)
The attribution method provides a direction for interpreting opaque neural networks in a visual way by identifying and visualizing the input regions/pixels that dominate the output of a network. Regarding the attribution method for visually explaining video understanding networks, it is challenging because of the unique spatiotemporal dependencies existing in video inputs and the special 3D convolutional or recurrent structures of video understanding networks. However, most existing attribution methods focus on explaining networks taking a single image as input and a few works specifically devised for video attribution come short of dealing with diversified structures of video understanding networks. In this paper, we investigate a generic perturbation-based attribution method that is compatible with diversified video understanding networks. Besides, we propose a novel regularization term to enhance the method by constraining the smoothness of its attribution results in both spatial and temporal dimensions. In order to assess the effectiveness of different video attribution methods without relying on manual judgement, we introduce reliable objective metrics which are checked by a newly proposed reliability measurement. We verified the effectiveness of our method by both subjective and objective evaluation and comparison with multiple significant attribution methods.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 1 Sep 2021.
Builds on
47 of 71 references.
References · 47
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2013Intriguing properties of neural networkscited by 5,155
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2015VQA: Visual Question Answeringcited by 2,506
- 2015Learning Deep Features for Discriminative Localizationcited by 2,436
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2016Adversarial examples in the physical worldcited by 1,769
35 more
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
- 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
- 2017SmoothGrad: removing noise by adding noisecited by 1,109
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
- 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
- 2014Striving for Simplicity: The All Convolutional Netcited by 990
- 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
- 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
- 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
- 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
- 2017Dense-Captioning Events in Videoscited by 793
- 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
- 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
- 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
- 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
- 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
- 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
- 2016Top-down Neural Attention by Excitation Backpropcited by 321
- 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 259
- 2017Real Time Image Saliency for Black Box Classifierscited by 246
- 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
- 2017Interpretable Convolutional Neural Networkscited by 162
- 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
- 2019XRAI: Better Attributions Through Regionscited by 116
- 2018Motion-Appearance Co-Memory Networks for Video Question Answeringcited by 95
- 2020Attribution in Scale and Spacecited by 61
- 2018Review of Visual Saliency Detection with Comprehensive Informationcited by 60
- 2020There and Back Again: Revisiting Backpropagation Saliency Methodscited by 60
- 2019Visualizing Deep Networks by Optimizing with Integrated Gradientscited by 59
- 2016Investigating the influence of noise and distractors on the interpretation of neural networkscited by 54
- 2019Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networkscited by 49
- 2020IROF: a low resource evaluation metric for explanation methodscited by 46
- 2019Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutionscited by 20
- 2017Excitation Backprop for RNNscited by 14
- 2019Class Feature Pyramids for Video Explanationcited by 8
Built on by
Citing papers per year: 2022: 3.
3 papers.