mlsanityData to 13 Oct 2026

StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

arXiv 2411.03628 ↗First posted 6 Nov 2024cs.CV cs.AIcited by 138

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on offline video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a significant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: (1) real-time visual understanding, (2) omni-source understanding, and (3) contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features five questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 13 open-source and proprietary MLLMs and find that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform significantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.

Key facts. Cited by 138 papers in the map (data through 13 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 30 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

27 of 30 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  2. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos26 Aug
  3. 2024Long Context Transfer from Language to Vision24 Jun
  4. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 19

  1. 2023Visual Instruction Tuningcited by 7,827
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,535
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 680
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 576
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 560
  11. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 387
  12. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
7 more
  1. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  2. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 283
  3. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 220
  4. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  5. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 163
  6. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  7. 2024VILA2: VILA Augmented VILAcited by 25

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 37; 2026: 98.

138 papers.

Sort:

By area:

  1. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 148
  2. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 137
  3. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 93
  4. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 73
  5. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 60
  6. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 55
  7. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 51
  8. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 49
  9. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  10. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 48
  11. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 46
  12. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 42
38 more
  1. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 39
  2. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  3. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 37
  4. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 35
  5. 2026A Simple Baseline for Streaming Video Understandingcited by 30
  6. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  7. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 27
  8. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 27
  9. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 25
  10. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 25
  11. 2024StreamChat: Chatting with Streaming Videocited by 23
  12. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  13. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 21
  14. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  15. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 20
  16. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 20
  17. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  18. 2026OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoningcited by 17
  19. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 16
  20. 2026Thinking in Streaming Videocited by 14
  21. 2025OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputscited by 13
  22. 2026RIVER: A Real-Time Interaction Benchmark for Video LLMscited by 13
  23. 2025TeleEgo: Benchmarking Egocentric AI Assistants in the Wildcited by 11
  24. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 11
  25. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  26. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  27. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 10
  28. 2025HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMscited by 9
  29. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  30. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 9
  31. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 9
  32. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 9
  33. 2025LiveCC: Learning Video LLM with Streaming Speech Transcription at Scalecited by 8
  34. 2025StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videoscited by 8
  35. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 8
  36. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  37. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 8
  38. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 7

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
  2. 2026FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
  3. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  4. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMs
  5. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  6. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  7. 2025An Empirical Study for Representations of Videos in Video Question Answering via MLLMs
  8. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streams