mlsanityData to 7 Oct 2026

Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

arXiv 2106.05392 ↗First posted 9 Jun 2021cs.CVcited by 224

Not on the map. Why?

Abstract (arXiv)

In video transformers, the time dimension is often treated in the same way as the two spatial dimensions. However, in a scene where objects or the camera may move, a physical point imaged at one location in frame t may be entirely unrelated to what is found at that location in frame t+k. These temporal correspondences should be modeled to facilitate learning about dynamic scenes. To this end, we propose a new drop-in block for video transformers -- trajectory attention -- that aggregates information along implicitly determined motion paths. We additionally propose a new method to address the quadratic dependence of computation and memory on the input size, which is particularly important for high resolution or long videos. While these ideas are useful in a range of settings, we apply them to the specific task of video action recognition with a transformer model and obtain state-of-the-art results on the Kinetics, Something--Something V2, and Epic-Kitchens datasets. Code and models are available at: https://github.com/facebookresearch/Motionformer

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Builds on 80 of its 107 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

80 of 107 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016Layer Normalizationcited by 5,579
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2020Rethinking Attention with Performerscited by 1,584
  13. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  14. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  15. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  16. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  17. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  18. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  19. 2017Mixed Precision Trainingcited by 1,135
  20. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  21. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  22. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  23. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  24. 2021Multiscale Vision Transformerscited by 896
  25. 2020Object-Centric Learning with Slot Attentioncited by 865
  26. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  27. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  28. 2018Image Transformercited by 770
  29. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  30. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  31. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  32. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  33. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  34. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  35. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  36. 2020Multi-modal Transformer for Video Retrievalcited by 395
  37. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  38. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 63; 2023: 59; 2024: 47; 2025: 29; 2026: 12.

224 papers.

Sort:

By area:

  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  2. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  3. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  4. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  5. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  6. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  7. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  8. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  9. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  10. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  11. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  12. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
38 more
  1. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  2. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  3. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  4. 2022Multiview Transformers for Video Recognitioncited by 148
  5. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  6. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  7. 2022Towards An End-to-End Framework for Flow-Guided Video Inpaintingcited by 111
  8. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  9. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  10. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  11. 2021SeqFormer: Sequential Transformer for Video Instance Segmentationcited by 98
  12. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  13. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  14. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  15. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  16. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  17. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  18. 2023An Outlook into the Future of Egocentric Visioncited by 84
  19. 2022TALLFormer: Temporal Action Localization with a Long-memory Transformercited by 78
  20. 2021Object-Region Video Transformerscited by 73
  21. 2022Video Transformers: A Surveycited by 63
  22. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  23. 2022Learning Trajectory-Aware Transformer for Video Super-Resolutioncited by 61
  24. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 58
  25. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  26. 2022Future Transformer for Long-term Action Anticipationcited by 57
  27. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  28. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  29. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  30. 2022InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challengescited by 48
  31. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  32. 2022Audiovisual Masked Autoencoderscited by 47
  33. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 44
  34. 2023Annotation-free Audio-Visual Segmentationcited by 44
  35. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 43
  36. 2021Long-Short Temporal Contrastive Learning of Video Transformerscited by 42
  37. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  38. 2022Deep Frequency Filtering for Domain Generalizationcited by 41

50 most cited of 224 All 224 (JSON)