mlsanityData to 7 Oct 2026

MovieChat+: Question-aware Sparse Memory for Long Video Question Answering

arXiv 2404.17176 ↗First posted 26 Apr 2024cs.CVcited by 73

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing methods either employ complex spatial-temporal modules or rely heavily on additional perception models to extract temporal features for video understanding, and they only perform well on short videos. For long videos, the computational complexity and memory costs associated with long-term temporal connections are significantly increased, posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose MovieChat to overcome these challenges. We lift pre-trained multi-modal large language models for understanding long videos without incorporating additional trainable temporal modules, employing a zero-shot approach. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video, 2K temporal grounding labels, and 14K manual annotations for validation of the effectiveness of our method. The code along with the dataset can be accessed via the following https://github.com/rese1f/MovieChat.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 74 of its 101 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

74 of 101 references.

References · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  7. 2021Video Swin Transformercited by 1,090
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2022Token Merging: Your ViT But Fastercited by 995
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  13. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  15. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  16. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  17. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  18. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  19. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  20. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  21. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  22. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  23. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  24. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  25. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  26. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  27. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  28. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  29. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  30. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  31. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  32. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  33. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  34. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  35. 2021Towards Long-Form Video Understandingcited by 146
  36. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  37. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  38. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 31; 2026: 22.

73 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  3. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  4. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  5. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  6. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  7. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  8. 2024SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memorycited by 91
  9. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  10. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  11. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  12. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
38 more
  1. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 58
  2. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  3. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  4. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  5. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  6. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  7. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  8. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  9. 2024CityCraft: A Real Crafter for 3D City Generationcited by 25
  10. 2024Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraintcited by 24
  11. 2024StreamChat: Chatting with Streaming Videocited by 23
  12. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  13. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  14. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 18
  15. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  16. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 15
  17. 2024Enhancing Long Video Understanding via Hierarchical Event-Based Memorycited by 14
  18. 2025∞-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidationcited by 14
  19. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  20. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  21. 2024SEAL: Semantic Attention Learning for Long Video Representationcited by 12
  22. 2025Question-Aware Gaussian Experts for Audio-Visual Question Answeringcited by 12
  23. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  24. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  25. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 10
  26. 2024Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Modelscited by 9
  27. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Modelscited by 9
  28. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  29. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  30. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  31. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  32. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  33. 2025NeuS-QA: Grounding Long-Form Video Understanding in Temporal Logic and Neuro-Symbolic Reasoningcited by 7
  34. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  35. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understandingcited by 6
  36. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 6
  37. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  38. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  4. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignment
  5. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  6. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
  7. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  8. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models