mlsanityData to 7 Oct 2026

StreamingVLM: Real-Time Understanding for Infinite Video Streams

arXiv 2510.09608 ↗First posted 10 Oct 2025cs.CV cs.AI cs.CLcited by 89

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on LongVideoBench by +4.30 and OVOBench Realtime by +5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 16 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

11 of 16 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  2. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov

Other references · 7

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  5. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  6. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  7. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 80.

89 papers.

Sort:

By area:

  1. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  2. 2025VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memorycited by 39
  3. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  4. 2026A Simple Baseline for Streaming Video Understandingcited by 29
  5. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  6. 2025Streaming Video Instruction Tuningcited by 25
  7. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  8. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  9. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 19
  10. 2026LongStream: Long-Sequence Streaming Autoregressive Visual Geometrycited by 17
  11. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 15
  12. 2026Thinking in Streaming Videocited by 14
38 more
  1. 2026Proact-VL: A Proactive VideoLLM for Real-Time AI Companionscited by 12
  2. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understandingcited by 11
  3. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  4. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 10
  5. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  6. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  7. 2026ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoningcited by 8
  8. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  9. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 7
  10. 2026What Should a Streaming Video Model Remember?cited by 7
  11. 2026Event-VStream: Event-Driven Real-Time Understanding for Long Video Streamscited by 6
  12. 2026Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Modelscited by 6
  13. 2026SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenancecited by 6
  14. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  15. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 6
  16. 2026CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Managementcited by 5
  17. 2026StreamingClaw Technical Reportcited by 4
  18. 2026OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMscited by 4
  19. 2026ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understandingcited by 4
  20. 2026See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoningcited by 3
  21. 2026StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Videocited by 3
  22. 2026EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Visioncited by 3
  23. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  24. 2026Harnessing Streaming Video in the Wildcited by 3
  25. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  26. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 3
  27. 2026StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMscited by 3
  28. 2026FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understandingcited by 2
  29. 2026STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understandingcited by 2
  30. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understandingcited by 2
  31. 2026GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstructioncited by 2
  32. 2026Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interactioncited by 2
  33. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 2
  34. 2025Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigationcited by 1
  35. 2025CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusioncited by 1
  36. 2026VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Modelscited by 1
  37. 2026CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understandingcited by 1
  38. 2026SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentarycited by 1

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
  2. 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
  3. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
  4. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memory
  5. 2025VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers
  6. 2026Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models
  7. 2025Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models
  8. 2026ReMem: Streaming Video Understanding With Long Context Retention