mlsanityData to 14 Oct 2026

LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval

arXiv 2505.15269 ↗First posted 21 May 2025cs.CVcited by 54

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Recent developments in Video Large Language Models (Video LLMs) have enabled models to process hour-long videos and exhibit exceptional performance. Nonetheless, the Key-Value (KV) cache expands linearly over time, leading to substantial memory overhead and response delay--critical challenges in various real-world online applications, such as Deepseek services, autonomous driving and robotics. To mitigate these issues, we propose LiveVLM, a training-free and query-agnostic framework specifically designed for online video understanding and real-time interaction. LiveVLM employs a Vision Sink Bucketing (VSB) mechanism to process video streams in real time, retain long-term video details and eliminate redundant KVs. This mechanism utilizes vision-to-vision attention scores as the metric and seeks to maximize the coverage of contextual information during compression. Noting that KV cache compressed in a query-agnostic manner inevitably retains irrelevant information for specific queries, LiveVLM incorporates a Position-agnostic KV Retrieval (PaR) mechanism to reduce interference from redundant context. The keypoint of PaR lies in decoupling positional embeddings to enhance the similarity between key tensors, thereby supporting efficient retrieval at the granularity of pages. Extensive experiments demonstrate that LiveVLM enables the foundation LLaVA-OneVision model to achieve state-of-the-art accuracy among both training-free query-agnostic methods and training-based online models.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 53 of its 63 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

53 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrieval1 Mar
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
38 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  8. 2025Qwen2.5-Omni Technical Reportcited by 736
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  13. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  14. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  15. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  16. 2024Long Context Transfer from Language to Visioncited by 515
  17. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  18. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  19. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  20. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  21. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  22. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  23. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  24. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  25. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  26. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  27. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  28. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  29. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  30. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  31. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  32. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  33. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  34. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 98
  35. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  36. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  37. 2022Learning to Merge Tokens in Vision Transformerscited by 65
  38. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 47.

54 papers.

Sort:

By area:

  1. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 58
  2. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  3. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 37
  4. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  5. 2026A Simple Baseline for Streaming Video Understandingcited by 33
  6. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 28
  7. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 22
  8. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  9. 2026STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstructioncited by 9
  10. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 9
  11. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  12. 2026StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understandingcited by 9
38 more
  1. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 8
  2. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  3. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 7
  4. 2026What Should a Streaming Video Model Remember?cited by 7
  5. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 6
  6. 2026Event-VStream: Event-Driven Real-Time Understanding for Long Video Streamscited by 6
  7. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  8. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrievalcited by 5
  9. 2026Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queriescited by 5
  10. 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inferencecited by 5
  11. 2026StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggeringcited by 5
  12. 2026StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMscited by 5
  13. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  14. 2026STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understandingcited by 4
  15. 2026Don't Pause! Every prediction matters in a streaming videocited by 4
  16. 2026An Efficient Streaming Video Understanding Framework with Agentic Controlcited by 4
  17. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  18. 2026Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understandingcited by 4
  19. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4
  20. 2026StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understandingcited by 3
  21. 2026FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understandingcited by 2
  22. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understandingcited by 2
  23. 2026ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandingcited by 2
  24. 2026Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memorycited by 1
  25. 2026Linear Scaling Video VLMs for Long Video Understandingcited by 1
  26. 2026Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?cited by 1
  27. 2026Think in Sets for Streaming Video Token Compressioncited by 1
  28. 2026StreamFlow: Dynamic Memory Flows for Streaming Video Understandingcited by 1
  29. 2026Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understandingcited by 1
  30. 2026Color When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensingcited by 0
  31. 2026Online Reasoning Video Object Segmentationcited by 0
  32. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Modelscited by 0
  33. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0
  34. 2026PREM: Prefix-Steered Recurrent Memory for Long-Video Understandingcited by 0
  35. 2026Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streamscited by 0
  36. 2026DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Videocited by 0
  37. 2026KeyRec: Bounded Visual Memory for Streaming and Long-Video Understandingcited by 0
  38. 2026Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Modelscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streams
  2. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding
  3. 2025CacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding
  4. 2026CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understanding
  5. 2025VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers
  6. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understanding
  7. 2026Small Vision-Language Models are Smart Compressors for Long Video Understanding
  8. 2026LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams