mlsanityData to 7 Oct 2026

Is Space-Time Attention All You Need for Video Understanding?

arXiv 2102.05095 ↗First posted 9 Feb 2021cs.CVcited by 1,609

Not on the map. Why?

Cited by 64 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.

Key facts. Cited by 1,609 papers in the map (data through 7 Oct 2026). Builds on 60 of its 73 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

60 of 73 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  12. 2017Non-local Neural Networkscited by 2,290
38 more
  1. 2018SlowFast Networks for Video Recognitioncited by 1,862
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  9. 2018Image Transformercited by 770
  10. 2020Quantifying Attention Flow in Transformerscited by 759
  11. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  13. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  14. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  15. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  16. 2019Attention Augmented Convolutional Networkscited by 363
  17. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  18. 2019Axial Attention in Multidimensional Transformerscited by 350
  19. 2018A Short Note about Kinetics-600cited by 331
  20. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  21. 2018Video Action Transformer Networkcited by 316
  22. 2020Exploring Self-attention for Image Recognitioncited by 313
  23. 2017Relation Networks for Object Detectioncited by 305
  24. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  25. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  26. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  27. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  28. 2018Scaling Neural Machine Translationcited by 204
  29. 2019Scaling Autoregressive Video Modelscited by 189
  30. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  31. 2019STM: SpatioTemporal and Motion Encoding for Action Recognitioncited by 142
  32. 2019Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolutioncited by 140
  33. 2018A2-Nets: Double Attention Networkscited by 136
  34. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  35. 2018Graph-Based Global Reasoning Networkscited by 120
  36. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  37. 2020Actor-Transformers for Group Activity Recognitioncited by 84
  38. 2017ActionVLAD: Learning spatio-temporal aggregation for action classificationcited by 73

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 132; 2022: 305; 2023: 319; 2024: 313; 2025: 309; 2026: 231.

1,609 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2021Video Swin Transformercited by 1,090
  7. 2021Multiscale Vision Transformerscited by 896
  8. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  9. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  11. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
38 more
  1. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  2. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  3. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  4. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  5. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  6. 2023CoTracker: It is Better to Track Togethercited by 516
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  9. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  10. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  11. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  12. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  13. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  14. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  15. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  16. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  17. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  18. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  19. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  20. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  21. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  22. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  23. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  24. 2022Egocentric Video-Language Pretrainingcited by 231
  25. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  26. 2023Spherical Fourier Neural Operators: Learning Stable Dynamics on the Spherecited by 225
  27. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  28. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  29. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  30. 2022VRT: A Video Restoration Transformercited by 219
  31. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  32. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  33. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  34. 2022Learning Video Representations from Large Language Modelscited by 204
  35. 2023A Survey on Video Diffusion Modelscited by 203
  36. 2023Video Understanding with Large Language Models: A Surveycited by 203
  37. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  38. 2021Anticipative Video Transformercited by 197

50 most cited of 1,609 All 1,609 (JSON)