mlsanityData to 14 Oct 2026

ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition

arXiv 2110.15574 ↗First posted 29 Oct 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

It is difficult for people to interpret the decision-making in the inference process of deep neural networks. Visual explanation is one method for interpreting the decision-making of deep learning. It analyzes the decision-making of 2D CNNs by visualizing an attention map that highlights discriminative regions. Visual explanation for interpreting the decision-making process in video recognition is more difficult because it is necessary to consider not only spatial but also temporal information, which is different from the case of still images. In this paper, we propose a visual explanation method called spatio-temporal attention branch network (ST-ABN) for video recognition. It enables visual explanation for both spatial and temporal information. ST-ABN acquires the importance of spatial and temporal information during network inference and applies it to recognition processing to improve recognition performance and visual explainability. Experimental results with Something-Something datasets V1 & V2 demonstrated that ST-ABN enables visual explanation that takes into account spatial and temporal information simultaneously and improves recognition performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 59 of its 72 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

59 of 72 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2017Squeeze-and-Excitation Networkscited by 4,789
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  9. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  10. 2017Non-local Neural Networkscited by 2,293
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  12. 2018SlowFast Networks for Video Recognitioncited by 1,867
38 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  2. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  3. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  5. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  7. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  8. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
  9. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  10. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  11. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  12. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  13. 2013Network In Networkcited by 605
  14. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  15. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  16. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
  17. 2017Residual Attention Network for Image Classificationcited by 386
  18. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  19. 2014Recurrent Models of Visual Attentioncited by 349
  20. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 345
  21. 2017Temporal Relational Reasoning in Videoscited by 324
  22. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  23. 2016Image Captioning with Semantic Attentioncited by 260
  24. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 259
  25. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  26. 2018Videos as Space-Time Region Graphscited by 221
  27. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  28. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  29. 2019STM: SpatioTemporal and Motion Encoding for Action Recognitioncited by 142
  30. 2018A2-Nets: Double Attention Networkscited by 136
  31. 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
  32. 2018Attention Branch Network: Learning of Attention Mechanism for Visual Explanationcited by 109
  33. 2018Learn To Pay Attentioncited by 83
  34. 2017ActionVLAD: Learning spatio-temporal aggregation for action classificationcited by 73
  35. 2019More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregationcited by 64
  36. 2019Grouped Spatial-Temporal Aggregation for Efficient Action Recognitioncited by 55
  37. 2017Attentional Pooling for Action Recognitioncited by 54
  38. 2015Action Recognition using Visual Attentioncited by 49

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.