mlsanityData to 14 Oct 2026

Long-VMNet: Accelerating Long-Form Video Understanding via Fixed Memory

arXiv 2503.13707 ↗First posted 17 Mar 2025cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Long-form video understanding is essential for various applications such as video retrieval, summarizing, and question answering. Yet, traditional approaches demand substantial computing power and are often bottlenecked by GPU memory. To tackle this issue, we present Long-Video Memory Network, Long-VMNet, a novel video understanding method that employs a fixed-size memory representation to store discriminative patches sampled from the input video. Long-VMNet achieves improved efficiency by leveraging a neural sampler that identifies discriminative tokens. Additionally, Long-VMNet only needs one scan through the video, greatly boosting efficiency. Our results on the Rest-ADL dataset demonstrate an 18x -- 75x improvement in inference times for long-form video retrieval and answering questions, with a competitive predictive performance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 48 of its 59 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

48 of 59 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  8. 2021ViViT: A Video Vision Transformercited by 1,620
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  11. 2021Video Swin Transformercited by 1,091
  12. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
36 more
  1. 2022Token Merging: Your ViT But Fastercited by 1,021
  2. 2021Multiscale Vision Transformerscited by 897
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  9. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  10. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  11. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  12. 2018Video Action Transformer Networkcited by 316
  13. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  14. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  15. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  16. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  17. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  18. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  19. 2024An Introduction to Vision-Language Modelingcited by 178
  20. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  21. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  22. 2021Towards Long-Form Video Understandingcited by 146
  23. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  24. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 98
  25. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  26. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  27. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 70
  28. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 59
  29. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  30. 2017Attend and Interact: Higher-Order Object Interactions for Video Understandingcited by 37
  31. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  32. 2019Tripping through time: Efficient Localization of Activities in Videoscited by 33
  33. 2021PyTorchVideo: A Deep Learning Library for Video Understandingcited by 32
  34. 2022Token Turing Machinescited by 25
  35. 2024Enhancing Long Video Understanding via Hierarchical Event-Based Memorycited by 14
  36. 2023Are current long-term video understanding datasets long-term?cited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  3. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
  4. 2026ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inferencecited by 1