mlsanityData to 14 Oct 2026

VideoLLM-online: Online Video Large Language Model for Streaming Video

arXiv 2406.11816 ↗First posted 17 Jun 2024cs.CVcited by 225

Not on the map. Why?

Cited by 71 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent Large Language Models have been enhanced with vision capabilities, enabling them to comprehend images, videos, and interleaved vision-language content. However, the learning methods of these large multimodal models typically treat videos as predetermined clips, making them less effective and efficient at handling streaming video inputs. In this paper, we propose a novel Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned, long-context, and real-time conversation within a continuous video stream. Our LIVE framework comprises comprehensive approaches to achieve video streaming dialogue, encompassing: (1) a training objective designed to perform language modeling for continuous streaming inputs, (2) a data generation scheme that converts offline temporal annotations into a streaming dialogue format, and (3) an optimized inference pipeline to speed up the model responses in real-world video streams. With our LIVE framework, we built VideoLLM-online model upon Llama-2/Llama-3 and demonstrate its significant advantages in processing streaming videos. For instance, on average, our model can support streaming dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it also showcases state-of-the-art performance on public offline video benchmarks, such as recognition, captioning, and forecasting. The code, model, data, and demo have been made available at https://showlab.github.io/videollm-online.

Key facts. Cited by 225 papers in the map (data through 14 Oct 2026). Builds on 85 of its 99 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

85 of 99 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2020Scaling Laws for Neural Language Modelscited by 6,586
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  13. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  16. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  17. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  19. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  20. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  21. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  22. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  23. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  24. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  25. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  26. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  27. 2020Rescaling Egocentric Visioncited by 540
  28. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  29. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  30. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  31. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  32. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  33. 2016Visual Dialogcited by 377
  34. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  35. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  36. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  37. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  38. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 76; 2026: 132.

225 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  3. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  4. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  5. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  6. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  7. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  8. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  9. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  10. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  11. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  12. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
38 more
  1. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  2. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 61
  3. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  4. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  5. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 58
  6. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  7. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 57
  8. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  9. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 54
  10. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  11. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  12. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 47
  13. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 46
  14. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 43
  15. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  16. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  17. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  18. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 37
  19. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  20. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 29
  21. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 28
  22. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 28
  23. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 26
  24. 2025Streaming Video Instruction Tuningcited by 26
  25. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
  26. 2024Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackerscited by 24
  27. 2024StreamChat: Chatting with Streaming Videocited by 24
  28. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 22
  29. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 22
  30. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  31. 2025AI for Service: Proactive Assistance with AI Glassescited by 20
  32. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 19
  33. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  34. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  35. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  36. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 17
  37. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  38. 2026Thinking in Streaming Videocited by 16

50 most cited of 225 All 225 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Streaming Long Video Understanding with Large Language Models
  2. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understanding
  3. 2026FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
  4. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models
  5. 2026Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models
  6. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  7. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input
  8. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding