mlsanityData to 14 Oct 2026

Efficient Spatialtemporal Context Modeling for Action Recognition

arXiv 2103.11190 ↗First posted 20 Mar 2021cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Contextual information plays an important role in action recognition. Local operations have difficulty to model the relation between two elements with a long-distance interval. However, directly modeling the contextual information between any two points brings huge cost in computation and memory, especially for action recognition, where there is an additional temporal dimension. Inspired from 2D criss-cross attention used in segmentation task, we propose a recurrent 3D criss-cross attention (RCCA-3D) module to model the dense long-range spatiotemporal contextual information in video for action recognition. The global context is factorized into sparse relation maps. We model the relationship between points in the same line along the direction of horizon, vertical and depth at each time, which forms a 3D criss-cross structure, and duplicate the same operation with recurrent mechanism to transmit the relation between points in a line to a plane finally to the whole spatiotemporal space. Compared with the non-local method, the proposed RCCA-3D module reduces the number of parameters and FLOPs by 25% and 30% for video context modeling. We evaluate the performance of RCCA-3D with two latest action recognition networks on three datasets and make a thorough analysis of the architecture, obtaining the optimal way to factorize and fuse the relation maps. Comparisons with other state-of-the-art methods demonstrate the effectiveness and efficiency of our model.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 65 references that are in the map. First posted on arXiv 20 Mar 2021.

Builds on

38 of 65 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2017Squeeze-and-Excitation Networkscited by 4,789
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  8. 2017Non-local Neural Networkscited by 2,293
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2018SlowFast Networks for Video Recognitioncited by 1,867
  11. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  12. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
26 more
  1. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  2. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  3. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  4. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
  5. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  6. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  7. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  8. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
  9. 2017A simple neural network module for relational reasoningcited by 346
  10. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  11. 2017Temporal Relational Reasoning in Videoscited by 324
  12. 2018Video Action Transformer Networkcited by 316
  13. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  14. 2015Delving Deeper into Convolutional Networks for Learning Video Representationscited by 148
  15. 2019Expectation-Maximization Attention Networks for Semantic Segmentationcited by 147
  16. 2018A2-Nets: Double Attention Networkscited by 136
  17. 2017ConvNet Architecture Search for Spatiotemporal Feature Learningcited by 40
  18. 2018Spatio-Temporal Channel Correlation Networks for Action Classificationcited by 38
  19. 2018Compact Generalized Non-local Networkcited by 36
  20. 2020Richly Activated Graph Convolutional Network for Robust Skeleton-based Action Recognitioncited by 36
  21. 2018Multi-Fiber Networks for Video Recognitioncited by 34
  22. 2019Collaborative Spatio-temporal Feature Learning for Video Action Recognitioncited by 24
  23. 2020Feedback Graph Convolutional Network for Skeleton-based Action Recognitioncited by 22
  24. 2017Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classificationcited by 16
  25. 2019Action recognition with spatial-temporal discriminative filter bankscited by 15
  26. 2018Temporal-Spatial Mapping for Action Recognitioncited by 8

Built on by

Cited by year

Citing papers per year: 2021: 1.

1 paper.

  1. 2021LIGAR: Lightweight General-purpose Action Recognitioncited by 0