mlsanityData to 14 Oct 2026

MovieChat: From Dense Token to Sparse Memory for Long Video Understanding

arXiv 2307.16449 ↗First posted 31 Jul 2023cs.CVcited by 542

Not on the map. Why?

Cited by 221 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing systems can only handle videos with very few frames. For long videos, the computation complexity, memory cost, and long-term temporal connection impose additional challenges. Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose the MovieChat to overcome these challenges. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video and 14K manual annotations for validation of the effectiveness of our method.

Key facts. Cited by 542 papers in the map (data through 14 Oct 2026). Builds on 93 of its 141 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

93 of 141 references.

References · 93

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  10. 2021Video Swin Transformercited by 1,091
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  13. 2022Token Merging: Your ViT But Fastercited by 1,021
  14. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  15. 2023A Survey on Multimodal Large Language Modelscited by 909
  16. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  18. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  19. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  20. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  21. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  22. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  23. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  24. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  25. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  26. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  27. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  28. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  29. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  30. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  31. 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
  32. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  33. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  34. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  35. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  36. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  37. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  38. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 135; 2025: 234; 2026: 159.

542 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  5. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  6. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  7. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  8. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  9. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  10. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  11. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  12. 20243D Reconstruction with Spatial Memorycited by 277
38 more
  1. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  2. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  3. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  4. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  5. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  6. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  7. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  8. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  9. 2023Video Understanding with Large Language Models: A Surveycited by 206
  10. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  11. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  12. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 178
  13. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  14. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  15. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  16. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  17. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  18. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  19. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  20. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  21. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  22. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  23. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  24. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 115
  25. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  26. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  27. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  28. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  29. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 106
  30. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  31. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  32. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 97
  33. 2024SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memorycited by 94
  34. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  35. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  36. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  37. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  38. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89

50 most cited of 542 All 542 (JSON)