mlsanityData to 7 Oct 2026

Streaming Long Video Understanding with Large Language Models

arXiv 2405.16009 ↗First posted 25 May 2024cs.CVcited by 164

Not on the map. Why?

Cited by 65 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents VideoStreaming, an advanced vision-language large model (VLLM) for video understanding, that capably understands arbitrary-length video with a constant number of video tokens streamingly encoded and adaptively selected. The challenge of video understanding in the vision language area mainly lies in the significant computational burden caused by the great number of tokens extracted from long videos. Previous works rely on sparse sampling or frame compression to reduce tokens. However, such approaches either disregard temporal information in a long time span or sacrifice spatial details, resulting in flawed compression. To address these limitations, our VideoStreaming has two core designs: Memory-Propagated Streaming Encoding and Adaptive Memory Selection. The Memory-Propagated Streaming Encoding architecture segments long videos into short clips and sequentially encodes each clip with a propagated memory. In each iteration, we utilize the encoded results of the preceding clip as historical memory, which is integrated with the current clip to distill a condensed representation that encapsulates the video content up to the current timestamp. After the encoding process, the Adaptive Memory Selection strategy selects a constant number of question-related memories from all the historical memories and feeds them into the LLM to generate informative responses. The question-related selection reduces redundancy within the memories, enabling efficient and precise video understanding. Meanwhile, the disentangled video extraction and reasoning design allows the LLM to answer different questions about a video by directly selecting corresponding memories, without the need to encode the whole video for each question. Our model achieves superior performance and higher efficiency on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Builds on 85 of its 99 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

85 of 99 references.

References · 85

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Mistral 7Bcited by 3,065
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2024Mixtral of Expertscited by 1,785
  13. 2020Reformer: The Efficient Transformercited by 1,658
  14. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  15. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  16. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  17. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  19. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  20. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  21. 2022Token Merging: Your ViT But Fastercited by 995
  22. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  23. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  24. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  25. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  26. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  27. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  28. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  29. 2023Textbooks Are All You Needcited by 561
  30. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  31. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  32. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  33. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  34. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  35. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  36. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  37. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  38. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 82; 2026: 66.

164 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  3. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  4. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  5. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  6. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  7. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  8. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  9. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  10. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  11. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 89
  12. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
38 more
  1. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  2. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  3. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  4. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  5. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  6. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  7. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 58
  8. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  9. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  10. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  11. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 55
  12. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  13. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  14. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  15. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  16. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  17. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  18. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  19. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  20. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  21. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  22. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  23. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  24. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  25. 2026A Simple Baseline for Streaming Video Understandingcited by 29
  26. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  27. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  28. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  29. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  30. 2024StreamChat: Chatting with Streaming Videocited by 23
  31. 2024RMem: Restricted Memory Banks Improve Video Object Segmentationcited by 22
  32. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  33. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22
  34. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  35. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  36. 2025FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learningcited by 20
  37. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  38. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19

50 most cited of 164 All 164 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ReMem: Streaming Video Understanding With Long Context Retention
  2. 2024ReWind: Understanding Long Videos with Instructed Learnable Memory
  3. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
  4. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
  5. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding
  6. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding
  7. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
  8. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models