mlsanityData to 7 Oct 2026

Higher Order Recurrent Space-Time Transformer for Video Action Prediction

arXiv 2104.08665 ↗First posted 17 Apr 2021cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Endowing visual agents with predictive capability is a key step towards video intelligence at scale. The predominant modeling paradigm for this is sequence learning, mostly implemented through LSTMs. Feed-forward Transformer architectures have replaced recurrent model designs in ML applications of language processing and also partly in computer vision. In this paper we investigate on the competitiveness of Transformer-style architectures for video predictive tasks. To do so we propose HORST, a novel higher order recurrent layer design whose core element is a spatial-temporal decomposition of self-attention for video. HORST achieves state of the art competitive performance on Something-Something early action recognition and EPIC-Kitchens action anticipation, showing evidence of predictive capability that we attribute to our recurrent higher order design of self-attention.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 17 Apr 2021.

Builds on

45 of 63 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
33 more
  1. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  2. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  3. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  4. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  5. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  6. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  7. 2018Video Action Transformer Networkcited by 316
  8. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  9. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  10. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218
  11. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  12. 2018A2-Nets: Double Attention Networkscited by 136
  13. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  14. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  15. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  16. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77
  17. 2019Gate-Shift Networks for Video Action Recognitioncited by 76
  18. 2017ActionVLAD: Learning spatio-temporal aggregation for action classificationcited by 73
  19. 2016VideoLSTM Convolves, Attends and Flows for Action Recognitioncited by 60
  20. 2018LSTA: Long Short-Term Attention for Egocentric Action Recognitioncited by 57
  21. 2017Attentional Pooling for Action Recognitioncited by 54
  22. 2020Convolutional Tensor-Train LSTM for Spatio-temporal Learningcited by 54
  23. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  24. 2018Smooth Loss Functions for Deep Top-k Classificationcited by 46
  25. 2015Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecturecited by 42
  26. 2021Learning to Anticipate Egocentric Actions by Imaginationcited by 40
  27. 2020Long-Term Anticipation of Activities with Cycle Consistencycited by 39
  28. 2019Uncertainty-Aware Anticipation of Activitiescited by 38
  29. 2017Encouraging LSTMs to Anticipate Actions Very Earlycited by 37
  30. 2017Long-term Forecasting using Higher Order Tensor RNNscited by 34
  31. 2016Higher Order Recurrent Neural Networkscited by 11
  32. 2021Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionariescited by 6
  33. 2019An Evaluation of Action Recognition Models on EPIC-Kitchenscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 1; 2024: 1.

6 papers.

  1. 2022Vision Transformers for Action Recognition: A Surveycited by 25
  2. 2022Unified Recurrence Modeling for Video Action Anticipationcited by 7
  3. 2024Uncertainty-boosted Robust Video Activity Anticipationcited by 7
  4. 2023VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipationcited by 4
  5. 2022NVIDIA-UNIBZ Submission for EPIC-KITCHENS-100 Action Anticipation Challenge 2022cited by 2
  6. 2022Inductive Attention for Video Action Anticipationcited by 1