mlsanityData to 14 Oct 2026

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

arXiv 2404.05726 ↗First posted 8 Apr 2024cs.CVcited by 229

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

With the success of large language models (LLMs), integrating the vision model into LLMs to build vision-language foundation models has gained much more interest recently. However, existing LLM-based large multimodal models (e.g., Video-LLaMA, VideoChat) can only take in a limited number of frames for short video understanding. In this study, we mainly focus on designing an efficient and effective model for long-term video understanding. Instead of trying to process more frames simultaneously like most existing work, we propose to process videos in an online manner and store past video information in a memory bank. This allows our model to reference historical video content for long-term analysis without exceeding LLMs' context length constraints or GPU memory limits. Our memory bank can be seamlessly integrated into current multimodal LLMs in an off-the-shelf manner. We conduct extensive experiments on various video understanding tasks, such as long-video understanding, video question answering, and video captioning, and our model can achieve state-of-the-art performances across multiple datasets. Code available at https://boheumd.github.io/MA-LMM/.

Key facts. Cited by 229 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 78 of its 92 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

78 of 92 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 77

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  2. 2018SlowFast Networks for Video Recognitioncited by 1,867
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  4. 2020Rethinking Attention with Performerscited by 1,601
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2022Token Merging: Your ViT But Fastercited by 1,021
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2021Multiscale Vision Transformerscited by 897
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  13. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  14. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  15. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  16. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  17. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 564
  18. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  19. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  20. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  21. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  22. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  23. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  24. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  25. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  26. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  27. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  28. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  29. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  30. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  31. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  32. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  33. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  34. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  35. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  36. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  37. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  38. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 39; 2025: 94; 2026: 96.

229 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  3. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  5. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  8. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  9. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  10. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  11. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  12. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
38 more
  1. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  2. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 97
  3. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  4. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  5. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 70
  6. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  7. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  8. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 58
  9. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  10. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 54
  11. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 53
  12. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  13. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  14. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  15. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 48
  16. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 46
  17. 2025MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narrativescited by 45
  18. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  19. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 38
  20. 2024Temporal Reasoning Transfer from Text to Videocited by 38
  21. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 38
  22. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 37
  23. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  24. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  25. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  26. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  27. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 28
  28. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  29. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  30. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 28
  31. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 28
  32. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 27
  33. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  34. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 25
  35. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
  36. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  37. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  38. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23

50 most cited of 229 All 229 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  2. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding
  3. 2024Understanding Long Videos with Multimodal Language Models
  4. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  5. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMs
  6. 2024Long Context Transfer from Language to Vision
  7. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  8. 2025Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders