mlsanityData to 14 Oct 2026

Memory Consolidation Enables Long-Context Video Understanding

arXiv 2402.05861 ↗First posted 8 Feb 2024cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

Most transformer-based video encoders are limited to short temporal contexts due to their quadratic complexity. While various attempts have been made to extend this context, this has often come at the cost of both conceptual and computational complexity. We propose to instead re-purpose existing pre-trained video transformers by simply fine-tuning them to attend to memories derived non-parametrically from past activations. By leveraging redundancy reduction, our memory-consolidated vision transformer (MC-ViT) effortlessly extends its context far into the past and exhibits excellent scaling behavior when learning from longer videos. In doing so, MC-ViT sets a new state-of-the-art in long-context video understanding on EgoSchema, Perception Test, and Diving48, outperforming methods that benefit from orders of magnitude more parameters.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 62 of its 72 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

62 of 72 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2016Layer Normalizationcited by 5,669
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2018SlowFast Networks for Video Recognitioncited by 1,867
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  2. 2022Visual Prompt Tuningcited by 1,681
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  4. 2021ViViT: A Video Vision Transformercited by 1,620
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2021Video Swin Transformercited by 1,091
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  13. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  14. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  15. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  16. 2014Neural Turing Machinescited by 644
  17. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  18. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  19. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  20. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  21. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  22. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  23. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  24. 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
  25. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  26. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  27. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  28. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  29. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  30. 2023Augmenting Language Models with Long-Term Memorycited by 189
  31. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  32. 2022Multiview Transformers for Video Recognitioncited by 148
  33. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  34. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  35. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  36. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  37. 2021Long Short-Term Transformer for Online Action Detectioncited by 88
  38. 2017Good Semi-supervised Learning that Requires a Bad GANcited by 86

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 18; 2026: 7.

45 papers.

Sort:

By area:

  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  2. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  3. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  4. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  5. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  6. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  7. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  8. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 65
  9. 2024VCA: Video Curious Agent for Long Video Understandingcited by 56
  10. 2024Language Repository for Long Video Understandingcited by 55
  11. 2024Foundation Models for Video Understanding: A Surveycited by 46
  12. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
33 more
  1. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 38
  2. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  3. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 27
  4. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  5. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 25
  6. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  7. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  8. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  9. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  10. 2025∞-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidationcited by 14
  11. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  12. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 8
  13. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  14. 2024Towards Long Video Understanding via Fine-detailed Video Story Generationcited by 6
  15. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understandingcited by 6
  16. 2026Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Modelscited by 6
  17. 2026VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memorycited by 6
  18. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  19. 2025Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understandingcited by 5
  20. 2025Moment Sampling in Video LLMs for Long-Form Video QAcited by 5
  21. 2024Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experiencecited by 4
  22. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 4
  23. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  24. 2024Extending Video Masked Autoencoders to 128 framescited by 3
  25. 2025Track-On2: Enhancing Online Point Tracking with Memorycited by 3
  26. 2026Towards Sparse Video Understanding and Reasoningcited by 2
  27. 2025Four Eyes Are Better Than Two: Harnessing the Collaborative Potential of Large Models via Differentiated Thinking and Complementary Ensemblescited by 1
  28. 2025See What You Need: Query-Aware Visual Intelligence through Reasoning-Perception Loopscited by 1
  29. 2025The Imperfect Learner: Incorporating Developmental Trajectories in Memory-based Student Simulationcited by 1
  30. 2025AdaVid: Adaptive Video-Language Pretrainingcited by 0
  31. 2025Temporal Cluster Assignment for Efficient Real-Time Video Segmentationcited by 0
  32. 2025Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extractioncited by 0
  33. 2026CAM: Question Answering on Entity-Centric Videos with Continuous Extraction and Adaptive Queryingcited by 0