# A Simple Baseline for Streaming Video Understanding

arXiv: https://arxiv.org/abs/2604.02317 · HTML: https://mlsanity.com/paper/2604.02317/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 48 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [Streaming Long Video Understanding with Large Language Models](https://mlsanity.com/paper/2405.16009/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025)
- [Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction](https://mlsanity.com/paper/2501.03218/) (2025)
- [Streaming Video Question-Answering with In-context Video KV-Cache Retrieval](https://mlsanity.com/paper/2503.00540/) (2025)
- [TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos](https://mlsanity.com/paper/2504.17343/) (2025)
- [StreamingVLM: Real-Time Understanding for Infinite Video Streams](https://mlsanity.com/paper/2510.09608/) (2025)
- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [StreamForest: Efficient Online Video Understanding with Persistent Event Memory](https://mlsanity.com/paper/2509.24871/) (2025)
- [Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge](https://mlsanity.com/paper/2501.13468/) (2025)
- [StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant](https://mlsanity.com/paper/2505.05467/) (2025)
- [Online Video Understanding: OVBench and VideoChat-Online](https://mlsanity.com/paper/2501.00584/) (2024)
- [StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding](https://mlsanity.com/paper/2508.15717/) (2025)
- [LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval](https://mlsanity.com/paper/2505.15269/) (2025)
- [OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts](https://mlsanity.com/paper/2503.22952/) (2025)
- [VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format](https://mlsanity.com/paper/2411.17991/) (2024)

## Built on by (most cited first, 30 of 33)

- [LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence](https://mlsanity.com/paper/2605.25979/) (2026)
- [Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model](https://mlsanity.com/paper/2607.24904/) (2026)
- [Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding](https://mlsanity.com/paper/2605.07897/) (2026)
- [What Should a Streaming Video Model Remember?](https://mlsanity.com/paper/2606.16353/) (2026)
- [OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs](https://mlsanity.com/paper/2606.03890/) (2026)
- [CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference](https://mlsanity.com/paper/2604.06036/) (2026)
- [ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding](https://mlsanity.com/paper/2607.28312/) (2026)
- [StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs](https://mlsanity.com/paper/2608.13416/) (2026)
- [OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants](https://mlsanity.com/paper/2605.26485/) (2026)
- [Harnessing Streaming Video in the Wild](https://mlsanity.com/paper/2606.08615/) (2026)
- [FOLIO: Focused Semantic Memory for Streaming Video Understanding](https://mlsanity.com/paper/2607.13298/) (2026)
- [Towards Spatial Supersensing in the Wild](https://mlsanity.com/paper/2607.13681/) (2026)
- [StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding](https://mlsanity.com/paper/2608.05703/) (2026)
- [StreamScout: Learning When to Look Deeper for Streaming Video Understanding](https://mlsanity.com/paper/2609.00291/) (2026)
- [SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models](https://mlsanity.com/paper/2609.05533/) (2026)
- [StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding](https://mlsanity.com/paper/2608.16320/) (2026)
- [ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding](https://mlsanity.com/paper/2609.02780/) (2026)
- [Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning](https://mlsanity.com/paper/2605.27318/) (2026)
- [Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?](https://mlsanity.com/paper/2606.09547/) (2026)
- [StreamFlow: Dynamic Memory Flows for Streaming Video Understanding](https://mlsanity.com/paper/2608.10949/) (2026)
- [Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding](https://mlsanity.com/paper/2609.04131/) (2026)
- [OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction](https://mlsanity.com/paper/2610.01762/) (2026)
- [VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Models](https://mlsanity.com/paper/2605.03351/) (2026)
- [DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video](https://mlsanity.com/paper/2609.27470/) (2026)
- [Simple Agentic Memory for Generalist Robot Policies](https://mlsanity.com/paper/2609.36595/) (2026)
- [When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs](https://mlsanity.com/paper/2609.37345/) (2026)
- [APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants](https://mlsanity.com/paper/2609.37559/) (2026)
- [FlashBack: Knowing When to Remember in Streaming Vision-Language Models](https://mlsanity.com/paper/2610.01192/) (2026)
- [ReMem: Streaming Video Understanding With Long Context Retention](https://mlsanity.com/paper/2610.05940/) (2026)
- [FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?](https://mlsanity.com/paper/2610.12427/) (2026)

## Abstract (arXiv)

Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.