mlsanityData to 7 Oct 2026

Anticipative Video Transformer

arXiv 2106.02036 ↗First posted 3 Jun 2021cs.CV cs.AI cs.LGcited by 197

Not on the map. Why?

Abstract (arXiv)

We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions. We train the model jointly to predict the next action in a video sequence, while also learning frame feature encoders that are predictive of successive future frames' features. Compared to existing temporal aggregation strategies, AVT has the advantage of both maintaining the sequential progression of observed actions while still capturing long-range dependencies--both critical for the anticipation task. Through extensive experiments, we show that AVT obtains the best reported performance on four popular action anticipation benchmarks: EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads; and it wins first place in the EpicKitchens-100 CVPR'21 challenge.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Builds on 71 of its 102 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

71 of 102 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2017Non-local Neural Networkscited by 2,290
38 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2017The Kinetics Human Action Video Datasetcited by 1,894
  3. 2018SlowFast Networks for Video Recognitioncited by 1,862
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2020Point Transformercited by 1,462
  8. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  13. 2021Multiscale Vision Transformerscited by 896
  14. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  15. 2020Quantifying Attention Flow in Transformerscited by 759
  16. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  17. 2017Look, Listen and Learncited by 452
  18. 2017Future Frame Prediction for Anomaly Detection -- A New Baselinecited by 391
  19. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  20. 2018Video Action Transformer Networkcited by 316
  21. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  22. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  23. 2021Video Transformer Networkcited by 271
  24. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  25. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  26. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  27. 2018Non-Local Recurrent Network for Image Restorationcited by 160
  28. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  29. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  30. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  31. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  32. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  33. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  34. 2021VidTr: Video Transformer Without Convolutionscited by 112
  35. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  36. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  37. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  38. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 43; 2023: 53; 2024: 39; 2025: 31; 2026: 24.

197 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  3. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  4. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  5. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  6. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  7. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  8. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  9. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  10. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  11. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  12. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
38 more
  1. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  2. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  3. 2023An Outlook into the Future of Egocentric Visioncited by 84
  4. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 79
  5. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  6. 2022Video Transformers: A Surveycited by 63
  7. 2023Grounded Question-Answering in Long Egocentric Videoscited by 62
  8. 2022Real-time Online Video Detection with Temporal Smoothing Transformerscited by 58
  9. 2022Future Transformer for Long-term Action Anticipationcited by 57
  10. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  11. 2023LoViT: Long Video Transformer for Surgical Phase Recognitioncited by 56
  12. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  13. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  14. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  15. 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
  16. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  17. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  18. 2023NaQ: Leveraging Narrations as Queries to Supervise Episodic Memorycited by 46
  19. 2022Visual Acoustic Matchingcited by 45
  20. 2023A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipationcited by 42
  21. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  22. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  23. 2022Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videoscited by 39
  24. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  25. 2023StepFormer: Self-supervised Step Discovery and Localization in Instructional Videoscited by 37
  26. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  27. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 36
  28. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  29. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  30. 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 32
  31. 2024Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognitioncited by 32
  32. 2022Egocentric Prediction of Action Target in 3Dcited by 31
  33. 2023WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognitioncited by 30
  34. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  35. 2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
  36. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  37. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  38. 2024Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mindcited by 28

50 most cited of 197 All 197 (JSON)