# ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding arXiv: https://arxiv.org/abs/2609.02780 ยท HTML: https://mlsanity.com/paper/2609.02780/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. - Builds on 52 of its 58 references that are in the map. - First posted on arXiv 2 Sep 2026. ## Placed in - Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/) ## Builds on (most cited first, 30 of 52) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025) - [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023) - [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025) - [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024) - [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024) - [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024) - [DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2503.02175/) (2025) - [Streaming Long Video Understanding with Large Language Models](https://mlsanity.com/paper/2405.16009/) (2024) - [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024) - [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025) - [Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction](https://mlsanity.com/paper/2501.03218/) (2025) - [Streaming Video Question-Answering with In-context Video KV-Cache Retrieval](https://mlsanity.com/paper/2503.00540/) (2025) - [TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos](https://mlsanity.com/paper/2504.17343/) (2025) - [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025) - [StreamForest: Efficient Online Video Understanding with Persistent Event Memory](https://mlsanity.com/paper/2509.24871/) (2025) - [StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant](https://mlsanity.com/paper/2505.05467/) (2025) - [SCBench: A KV Cache-Centric Analysis of Long-Context Methods](https://mlsanity.com/paper/2412.10319/) (2024) - [InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding](https://mlsanity.com/paper/2506.15745/) (2025) - [StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding](https://mlsanity.com/paper/2508.15717/) (2025) - [SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding](https://mlsanity.com/paper/2502.10810/) (2025) - [LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval](https://mlsanity.com/paper/2505.15269/) (2025) - [WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning](https://mlsanity.com/paper/2512.02425/) (2025) - [StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition](https://mlsanity.com/paper/2503.06220/) (2025) - [StreamingTOM: Streaming Token Compression for Efficient Video Understanding](https://mlsanity.com/paper/2510.18269/) (2025) - [ViSpeak: Visual Instruction Feedback in Streaming Videos](https://mlsanity.com/paper/2503.12769/) (2025) - [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2601.14724/) (2026) - [Accelerating Streaming Video Large Language Models via Hierarchical Token Compression](https://mlsanity.com/paper/2512.00891/) (2025) ## Built on by (most cited first, 2 of 2) - [When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs](https://mlsanity.com/paper/2609.37345/) (2026) - [FlashBack: Knowing When to Remember in Streaming Vision-Language Models](https://mlsanity.com/paper/2610.01192/) (2026) ## Abstract (arXiv) Streaming video understanding is a critical capability for real-world applications, including embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous video streams with multimodal large language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual token pruning, token merging, quantization, on-demand frame retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.