mlsanityData to 14 Oct 2026

HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

arXiv 2601.14724 ↗First posted 21 Jan 2026cs.CV cs.AI cs.CLcited by 41

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

also in 1 more
Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated significant improvement in offline video understanding. However, extending these capabilities to streaming video inputs, remains challenging, as existing models struggle to simultaneously maintain stable understanding performance, real-time responses, and low GPU memory overhead. To address this challenge, we propose HERMES, a novel training-free architecture for real-time and accurate understanding of video streams. Based on a mechanistic attention investigation, we conceptualize KV cache as a hierarchical memory framework that encapsulates video information across multiple granularities. During inference, HERMES reuses a compact KV cache, enabling efficient streaming understanding under resource constraints. Notably, HERMES requires no auxiliary computations upon the arrival of user queries, thereby guaranteeing real-time responses for continuous video stream interactions, which achieves 10× faster TTFT compared to prior SOTA. Even when reducing video tokens by up to 68% compared with uniform sampling, HERMES achieves superior or comparable accuracy across all benchmarks, with up to 11.4% gains on streaming datasets.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 55 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

41 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memory29 Sep
  2. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding21 Aug
  3. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding6 Nov
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  5. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  6. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  7. 2024Long Context Transfer from Language to Vision24 Jun
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov

Other references · 32

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  9. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  11. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  12. 2024Titans: Learning to Memorize at Test Timecited by 328
20 more
  1. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  2. 2025Memory in the Age of AI Agentscited by 303
  3. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  4. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  5. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  6. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  7. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  8. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  9. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  10. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 96
  11. 2025Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledgecited by 73
  12. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  13. 2025Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agentscited by 62
  14. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 58
  15. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  16. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  17. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 32
  18. 2025Cognitive Memory in Large Language Modelscited by 29
  19. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  20. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 41.

41 papers.

Sort:

By area:

  1. 2026A Simple Baseline for Streaming Video Understandingcited by 33
  2. 2026Thinking in Streaming Videocited by 16
  3. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  4. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 12
  5. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  6. 2026What Should a Streaming Video Model Remember?cited by 7
  7. 2026OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMscited by 6
  8. 2026CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Managementcited by 5
  9. 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inferencecited by 5
  10. 2026EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Visioncited by 5
  11. 2026ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understandingcited by 5
  12. 2026StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMscited by 5
29 more
  1. 2026OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMscited by 4
  2. 2026Harnessing Streaming Video in the Wildcited by 4
  3. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4
  4. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  5. 2026StreamScout: Learning When to Look Deeper for Streaming Video Understandingcited by 3
  6. 2026StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understandingcited by 2
  7. 2026ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandingcited by 2
  8. 2026Linear Scaling Video VLMs for Long Video Understandingcited by 1
  9. 2026ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inferencecited by 1
  10. 2026EgoPlay: Event-Triggered Video Editing for Egocentric Streamscited by 1
  11. 2026Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generationcited by 1
  12. 2026StreamFlow: Dynamic Memory Flows for Streaming Video Understandingcited by 1
  13. 2026Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understandingcited by 1
  14. 2026Omni-Streaming Thinkingcited by 1
  15. 2026VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Modelscited by 0
  16. 2026HY-Himmel Technical Report: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understandingcited by 0
  17. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Modelscited by 0
  18. 2026SLVMBench: Skill Learning from Video Memorycited by 0
  19. 2026ChronoStitch: Training-Free Composition of Visual KV Memories for Long-Horizon Temporal Reasoningcited by 0
  20. 2026DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Videocited by 0
  21. 2026KeyRec: Bounded Visual Memory for Streaming and Long-Video Understandingcited by 0
  22. 2026MemEvo: Automatic Discovery of Streaming Video Memory Mechanismscited by 0
  23. 2026When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMscited by 0
  24. 2026APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistantscited by 0
  25. 2026FlashBack: Knowing When to Remember in Streaming Vision-Language Modelscited by 0
  26. 2026ReMem: Streaming Video Understanding With Long Context Retentioncited by 0
  27. 2026SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understandingcited by 0
  28. 2026EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videoscited by 0
  29. 2026DeltaStreamer: Towards Human-Like Agentic Streaming Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory
  2. 2026ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory
  3. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
  4. 2026Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
  5. 2026LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams
  6. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understanding
  7. 2026CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understanding
  8. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval