mlsanityData to 7 Oct 2026

Two-Stream Transformer Architecture for Long Video Understanding

arXiv 2208.01753 ↗First posted 2 Aug 2022cs.CV cs.LG cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

Pure vision transformer architectures are highly effective for short video classification and action recognition tasks. However, due to the quadratic complexity of self attention and lack of inductive bias, transformers are resource intensive and suffer from data inefficiencies. Long form video understanding tasks amplify data and memory efficiency problems in transformers making current approaches unfeasible to implement on data or memory restricted domains. This paper introduces an efficient Spatio-Temporal Attention Network (STAN) which uses a two-stream transformer architecture to model dependencies between static image features and temporal contextual features. Our proposed approach can classify videos up to two minutes in length on a single GPU, is data efficient, and achieves SOTA performance on several long video understanding tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 47 of its 65 references that are in the map. First posted on arXiv 2 Aug 2022.

Builds on

47 of 65 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2018SlowFast Networks for Video Recognitioncited by 1,862
35 more
  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  3. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  4. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  5. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  6. 2021Multiscale Vision Transformerscited by 896
  7. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  8. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  9. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  10. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  11. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  12. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  13. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  14. 2017Residual Attention Network for Image Classificationcited by 385
  15. 2019Axial Attention in Multidimensional Transformerscited by 350
  16. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  17. 2018Video Action Transformer Networkcited by 316
  18. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  19. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  20. 2021Video Transformer Networkcited by 271
  21. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  22. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  23. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  24. 2021Anticipative Video Transformercited by 197
  25. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  26. 2021Towards Long-Form Video Understandingcited by 146
  27. 2021VidTr: Video Transformer Without Convolutionscited by 112
  28. 2020Actor-Context-Actor Relation Network for Spatio-Temporal Action Localizationcited by 91
  29. 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
  30. 2017Learnable pooling with Context Gating for video classificationcited by 58
  31. 2018Modeling Localness for Self-Attention Networkscited by 43
  32. 2017From Lifestyle Vlogs to Everyday Interactionscited by 42
  33. 2020AttentionNAS: Spatiotemporal Attention Cell Search for Video Classificationcited by 23
  34. 2015Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classificationcited by 21
  35. 2019Moviescope: Large-scale Analysis of Movies using Multiple Modalitiescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 1; 2025: 0; 2026: 2.

4 papers.

  1. 2022Vision Transformers for Action Recognition: A Surveycited by 25
  2. 2024Real-Time Human Action Recognition on Embedded Platformscited by 4
  3. 2026Two-Stream temporal transformer for video action classificationcited by 0
  4. 2026MSGL-Transformer: A Multi-Scale Global-Local Transformer for Rodent Social Behavior Recognitioncited by 0