mlsanityData to 14 Oct 2026

Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding

arXiv 2504.13915 ↗First posted 10 Apr 2025cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

We introduce ProVideLLM, an end-to-end framework for real-time procedural video understanding. ProVideLLM integrates a multimodal cache configured to store two types of tokens - verbalized text tokens, which provide compressed textual summaries of long-term observations, and visual tokens, encoded with DETR-QFormer to capture fine-grained details from short-term observations. This design reduces token count by 22x over existing methods in representing one hour of long-term observations while effectively encoding fine-granularity of the present. By interleaving these tokens in our multimodal cache, ProVideLLM ensures sub-linear scaling of memory and compute with video length, enabling per-frame streaming inference at 10 FPS and streaming dialogue at 25 FPS, with a minimal 2GB GPU memory footprint. ProVideLLM also sets new state-of-the-art results on six procedural tasks across four datasets.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 90 of its 102 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

90 of 102 references.

References · 90

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  13. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  14. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  15. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  17. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  19. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  20. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  21. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  22. 2020Rescaling Egocentric Visioncited by 540
  23. 2022Efficiently Scaling Transformer Inferencecited by 539
  24. 2024Long Context Transfer from Language to Visioncited by 515
  25. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  26. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  27. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  28. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  29. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  30. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  31. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  32. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  33. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  34. 2021Anticipative Video Transformercited by 200
  35. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  36. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  37. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  38. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  2. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  3. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  4. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  5. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  6. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 3
  7. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understandingcited by 2
  8. 2025SneakPeek: Future-Guided Instructional Streaming Video Generationcited by 1
  9. 2026On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understandingcited by 1
  10. 2026FlowNar: Scalable Streaming Narration for Long-Form Videoscited by 1
  11. 2026ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inferencecited by 1
  12. 2026ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memorycited by 1
7 more
  1. 2026Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turnscited by 1
  2. 2026QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understandingcited by 0
  3. 2026ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understandingcited by 0
  4. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0
  5. 2026Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challengescited by 0
  6. 2026Agentic Video Understanding: A Surveycited by 0
  7. 2026DeltaStreamer: Towards Human-Like Agentic Streaming Video Understandingcited by 0