# StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding

arXiv: https://arxiv.org/abs/2508.15717 · HTML: https://mlsanity.com/paper/2508.15717/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 57 of its 60 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 57)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding](https://mlsanity.com/paper/2404.05726/) (2024)
- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding](https://mlsanity.com/paper/2409.14485/) (2024)

## Built on by (most cited first, 30 of 58)

- [MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives](https://mlsanity.com/paper/2512.14699/) (2025)
- [StreamingTOM: Streaming Token Compression for Efficient Video Understanding](https://mlsanity.com/paper/2510.18269/) (2025)
- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2601.14724/) (2026)
- [Accelerating Streaming Video Large Language Models via Hierarchical Token Compression](https://mlsanity.com/paper/2512.00891/) (2025)
- [A Simple Baseline for Streaming Video Understanding](https://mlsanity.com/paper/2604.02317/) (2026)
- [FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding](https://mlsanity.com/paper/2603.02096/) (2026)
- [Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously](https://mlsanity.com/paper/2603.12262/) (2026)
- [StreamReady: Learning What to Answer and When in Long Streaming Videos](https://mlsanity.com/paper/2603.08620/) (2026)
- [Thinking in Streaming Video](https://mlsanity.com/paper/2603.12938/) (2026)
- [VideoNSA: Native Sparse Attention Scales Video Understanding](https://mlsanity.com/paper/2510.02295/) (2025)
- [STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstruction](https://mlsanity.com/paper/2603.20284/) (2026)
- [StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2604.09000/) (2026)
- [Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding](https://mlsanity.com/paper/2605.07897/) (2026)
- [What Should a Streaming Video Model Remember?](https://mlsanity.com/paper/2606.16353/) (2026)
- [Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2601.15655/) (2026)
- [MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering](https://mlsanity.com/paper/2605.22269/) (2026)
- [Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queries](https://mlsanity.com/paper/2602.08448/) (2026)
- [CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management](https://mlsanity.com/paper/2603.19571/) (2026)
- [CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference](https://mlsanity.com/paper/2604.06036/) (2026)
- [EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision](https://mlsanity.com/paper/2605.31557/) (2026)
- [StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs](https://mlsanity.com/paper/2608.13416/) (2026)
- [video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLM](https://mlsanity.com/paper/2510.11129/) (2025)
- [STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding](https://mlsanity.com/paper/2603.27593/) (2026)
- [Don't Pause! Every prediction matters in a streaming video](https://mlsanity.com/paper/2604.24317/) (2026)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs](https://mlsanity.com/paper/2606.07577/) (2026)
- [Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2606.25658/) (2026)
- [FOLIO: Focused Semantic Memory for Streaming Video Understanding](https://mlsanity.com/paper/2607.13298/) (2026)
- [Dynamic Hub-and-Spoke Memory for Streaming Video Understanding](https://mlsanity.com/paper/2608.30294/) (2026)
- [WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs](https://mlsanity.com/paper/2602.22142/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have made significant progress in visual-language reasoning, but their ability to efficiently handle long videos remains limited. Despite recent advances in long-context MLLMs, storing and attending to the key-value (KV) cache for long visual contexts incurs substantial memory and computational overhead. Existing visual compression methods require either encoding the entire visual context before compression or having access to the questions in advance, which is impractical for long video understanding and multi-turn conversational settings. In this work, we propose StreamMem, a query-agnostic KV cache memory mechanism for streaming video understanding. Specifically, StreamMem encodes new video frames in a streaming manner, compressing the KV cache using attention scores between visual tokens and generic query tokens, while maintaining a fixed-size KV memory to enable efficient question answering (QA) in memory-constrained, long-video scenarios. Evaluation on three long video understanding and two streaming video question answering benchmarks shows that StreamMem achieves state-of-the-art performance in query-agnostic KV cache compression and is competitive with query-aware compression approaches.