mlsanityData to 7 Oct 2026

Streaming Video Question-Answering with In-context Video KV-Cache Retrieval

arXiv 2503.00540 ↗First posted 1 Mar 2025cs.CVcited by 112

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

We propose ReKV, a novel training-free approach that enables efficient streaming video question-answering (StreamingVQA), by seamlessly integrating with existing Video Large Language Models (Video-LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding to queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon as user queries are received. Building on a common Video-LLM, we first incorporate a sliding-window attention mechanism, ensuring that input frames attend to a limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory as needed. Additionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video encoding and question-answering across different processes and GPUs, significantly enhancing the efficiency of StreamingVQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 53 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

46 of 53 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
34 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  5. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  8. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  9. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  10. 2024Long Context Transfer from Language to Visioncited by 502
  11. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  12. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  13. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  14. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  15. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  16. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  17. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  18. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  19. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  20. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  21. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  22. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  23. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  24. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  25. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  26. 2024Streaming Dense Video Captioningcited by 82
  27. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  28. 2023Grounded Question-Answering in Long Egocentric Videoscited by 62
  29. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  30. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  31. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 43
  32. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  33. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  34. 2024QuickLLaMA: Query-aware Inference Acceleration for Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 79.

112 papers.

Sort:

By area:

  1. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  2. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 89
  3. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  4. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  5. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  6. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  7. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  8. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  9. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  10. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  11. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  12. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
38 more
  1. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  2. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  3. 2026A Simple Baseline for Streaming Video Understandingcited by 29
  4. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  5. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  6. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 25
  7. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 24
  8. 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thoughtcited by 23
  9. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  10. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 21
  11. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  12. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 19
  13. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 18
  14. 2025VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carrierscited by 10
  15. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  16. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 10
  17. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 9
  18. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  19. 2026STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstructioncited by 8
  20. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 8
  21. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  22. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  23. 2025Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMscited by 7
  24. 2026Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Modelscited by 7
  25. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 7
  26. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 7
  27. 2026What Should a Streaming Video Model Remember?cited by 7
  28. 2026Event-VStream: Event-Driven Real-Time Understanding for Long Video Streamscited by 6
  29. 2026Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Modelscited by 6
  30. 2026Ego-Grounding for Personalized Question-Answering in Egocentric Videoscited by 6
  31. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  32. 2025CogStream: Context-guided Streaming Video Question Answeringcited by 5
  33. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 5
  34. 2026CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Managementcited by 5
  35. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  36. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrievalcited by 4
  37. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 4
  38. 2026StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggeringcited by 4

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding
  2. 2025A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering
  3. 2025Video Detective: Seek Critical Clues Recurrently to Answer Question from Long Videos
  4. 2026Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queries
  5. 2026Improving Video Question Answering through query-based frame selection
  6. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  7. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answering
  8. 2026Adaptive Greedy Frame Selection for Long Video Understanding