mlsanityData to 7 Oct 2026

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition

arXiv 2201.08383 ↗First posted 20 Jan 2022cs.CVcited by 189

Not on the map. Why?

Abstract (arXiv)

While today's video recognition systems parse snapshots or short clips accurately, they cannot connect the dots and reason across a longer range of time yet. Most existing video architectures can only process <5 seconds of a video without hitting the computation or memory bottlenecks. In this paper, we propose a new strategy to overcome this challenge. Instead of trying to process more frames at once like most existing methods, we propose to process videos in an online fashion and cache "memory" at each iteration. Through the memory, the model can reference prior context for long-term modeling, with only a marginal cost. Based on this idea, we build MeMViT, a Memory-augmented Multiscale Vision Transformer, that has a temporal support 30x longer than existing models with only 4.5% more compute; traditional methods need >3,000% more compute to do the same. On a wide range of settings, the increased temporal support enabled by MeMViT brings large gains in recognition accuracy consistently. MeMViT obtains state-of-the-art results on the AVA, EPIC-Kitchens-100 action classification, and action anticipation datasets. Code and models are available at https://github.com/facebookresearch/memvit.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Builds on 70 of its 90 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

70 of 90 references.

References · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2021ViViT: A Video Vision Transformercited by 1,610
38 more
  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2018Self-Attention with Relative Position Representationscited by 1,112
  4. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  5. 2020Designing Network Design Spacescited by 954
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  7. 2021Multiscale Vision Transformerscited by 896
  8. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2021Going deeper with Image Transformerscited by 759
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  13. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  14. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  15. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  16. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  17. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  18. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  19. 2018A Short Note about Kinetics-600cited by 331
  20. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  21. 2017Temporal Relational Reasoning in Videoscited by 324
  22. 2018Video Action Transformer Networkcited by 316
  23. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  24. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  25. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  26. 2021Video Transformer Networkcited by 271
  27. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  28. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  29. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  30. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  31. 2021Anticipative Video Transformercited by 197
  32. 2017Flow-Guided Feature Aggregation for Video Object Detectioncited by 177
  33. 2019Adaptive Attention Span in Transformerscited by 148
  34. 2021Towards Long-Form Video Understandingcited by 146
  35. 2019STM: SpatioTemporal and Motion Encoding for Action Recognitioncited by 142
  36. 2021Visformer: The Vision-friendly Transformercited by 137
  37. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  38. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 48; 2024: 69; 2025: 34; 2026: 16.

189 papers.

Sort:

By area:

  1. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  2. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  3. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  5. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  6. 2022Learning Video Representations from Large Language Modelscited by 204
  7. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  8. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  9. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  10. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  11. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  12. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
38 more
  1. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  2. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  3. 2023An Outlook into the Future of Egocentric Visioncited by 84
  4. 2024Streaming Dense Video Captioningcited by 82
  5. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  6. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  7. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  8. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  9. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  10. 2024Language Repository for Long Video Understandingcited by 55
  11. 2024MemFlow: Optical Flow Estimation and Prediction with Memorycited by 55
  12. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  13. 2022Retrieval-Enhanced Machine Learningcited by 46
  14. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  15. 2022Temporally Consistent Transformers for Video Generationcited by 43
  16. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  17. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  18. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  19. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 36
  20. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  21. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  22. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  23. 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 32
  24. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 31
  25. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  26. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  27. 2022Holistic Interaction Transformer Network for Action Detectioncited by 28
  28. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  29. 2022Token Turing Machinescited by 25
  30. 2023LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answeringcited by 25
  31. 2023Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulationcited by 23
  32. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  33. 2023Use Your Head: Improving Long-Tail Video Recognitioncited by 22
  34. 2024Video Token Merging for Long-form Video Understandingcited by 22
  35. 2023Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipationcited by 21
  36. 2023End-to-End Spatio-Temporal Action Localisation with Video Transformerscited by 20
  37. 2023Training a Large Video Model on a Single Machine in a Daycited by 20
  38. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20

50 most cited of 189 All 189 (JSON)