mlsanityData to 7 Oct 2026

Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams

arXiv 2406.08085 ↗First posted 12 Jun 2024cs.CVcited by 160

Not on the map. Why?

Cited by 59 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Benefiting from the advancements in large language models and cross-modal alignment, existing multi-modal video understanding methods have achieved prominent performance in offline scenario. However, online video streams, as one of the most common media forms in the real world, have seldom received attention. Compared to offline videos, the 'dynamic' nature of online video streams poses challenges for the direct application of existing models and introduces new problems, such as the storage of extremely long-term information, interaction between continuous visual content and 'asynchronous' user questions. Therefore, in this paper we present Flash-VStream, a video-language model that simulates the memory mechanism of human. Our model is able to process extremely long video streams in real-time and respond to user queries simultaneously. Compared to existing models, Flash-VStream achieves significant reductions in inference latency and VRAM consumption, which is intimately related to performing understanding of online streaming video. In addition, given that existing video understanding benchmarks predominantly concentrate on offline scenario, we propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding. Comparisons with popular existing methods on the proposed benchmark demonstrate the superiority of our method for such challenging setting. To verify the generalizability of our approach, we further evaluate it on existing video understanding benchmarks and achieves state-of-the-art performance in offline scenarios as well. All code, models, and datasets are available at the https://invinciblewyq.github.io/vstream-page/

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Builds on 44 of its 50 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

44 of 50 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
32 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  11. 2014Neural Turing Machinescited by 630
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  13. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  14. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  15. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  16. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  17. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  18. 2019Towards Real-Time Multi-Object Trackingcited by 311
  19. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  20. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  21. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  22. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  23. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 70
  24. 2018A Twofold Siamese Network for Real-Time Object Trackingcited by 62
  25. 2016Real-time Action Recognition with Enhanced Motion Vector CNNscited by 51
  26. 2019Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognitioncited by 35
  27. 2022Learning Quality-aware Dynamic Memory for Video Object Segmentationcited by 35
  28. 2023Universal Segmentation at Arbitrary Granularity with Language Instructioncited by 29
  29. 2019Distributed Deep Learning Model for Intelligent Video Surveillance Systems with Edge Computingcited by 28
  30. 2018A Memory-Network Based Solution for Multivariate Time-Series Forecastingcited by 21
  31. 2021Dynamic Memory based Attention Network for Sequential Recommendationcited by 16
  32. 2017Tracking as Online Decision-Making: Learning a Policy from Streaming Videos with Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 72; 2026: 71.

160 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  4. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  5. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  6. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  7. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  8. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  9. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  10. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  11. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  12. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
38 more
  1. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  2. 2025Scaling RL to Long Videoscited by 83
  3. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  4. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  5. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 58
  6. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  7. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  8. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 55
  9. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  10. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  11. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  12. 2025KV-Edit: Training-Free Image Editing for Precise Background Preservationcited by 47
  13. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  14. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  15. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  16. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  17. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  18. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  19. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  20. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  21. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  22. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 25
  23. 2025Streaming Video Instruction Tuningcited by 25
  24. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 24
  25. 2024StreamChat: Chatting with Streaming Videocited by 23
  26. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  27. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  28. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  29. 2025FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learningcited by 20
  30. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Controlcited by 19
  31. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  32. 2025Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Modelscited by 19
  33. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  34. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 18
  35. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  36. 2026OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoningcited by 16
  37. 2026From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agentscited by 14
  38. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12

50 most cited of 160 All 160 (JSON)