mlsanityData to 14 Oct 2026

SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding

arXiv 2502.10810 ↗First posted 15 Feb 2025cs.CVcited by 57

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Despite the significant advancements of Large Vision-Language Models (LVLMs) on established benchmarks, there remains a notable gap in suitable evaluation regarding their applicability in the emerging domain of long-context streaming video understanding. Current benchmarks for video understanding typically emphasize isolated single-instance text inputs and fail to evaluate the capacity to sustain temporal reasoning throughout the entire duration of video streams. To address these limitations, we introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains specifically designed to thoroughly assess the capabilities of streaming video understanding of current LVLMs. We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos, which includes generating QA chains that represent a series of consecutive multi-turn dialogues over video segments and constructing temporal linkages between successive QA chains. Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding. We also construct a StreamingChat model, which significantly outperforms open-source LVLMs on our SVBench and achieves comparable performance on diverse vision-language benchmarks. We expect SVBench to advance the research of streaming video understanding by providing a comprehensive and in-depth analysis of current LVLMs. Our benchmark and model can be accessed at https://github.com/sotayang/SVBench.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 59 references that are in the map. First posted on arXiv 15 Feb 2025.

Builds on

48 of 59 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  2. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  3. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  4. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  5. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  7. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  8. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  9. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
25 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  5. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  9. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  11. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  12. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  13. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  14. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  15. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  16. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  17. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  18. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  19. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  20. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  21. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  22. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  23. 2023Grounded Question-Answering in Long Egocentric Videoscited by 64
  24. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  25. 2023LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streamingcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 34.

57 papers.

Sort:

By area:

  1. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  2. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  3. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  4. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  5. 2025Streaming Video Instruction Tuningcited by 26
  6. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
  7. 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thoughtcited by 24
  8. 2026OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoningcited by 19
  9. 2025SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Modelscited by 18
  10. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 18
  11. 2026Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoningcited by 14
  12. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
38 more
  1. 2025StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videoscited by 8
  2. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  3. 2025MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialoguescited by 7
  4. 2026IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streamscited by 7
  5. 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interactioncited by 6
  6. 2025StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQAcited by 6
  7. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 6
  8. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 6
  9. 2025CogStream: Context-guided Streaming Video Question Answeringcited by 5
  10. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  11. 2025SoMe: A Realistic Benchmark for LLM-based Social Media Agentscited by 5
  12. 2025A Benchmark for Omni-Modal Reasoning in Long Videoscited by 5
  13. 2026LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Taskscited by 5
  14. 2026StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMscited by 5
  15. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  16. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4
  17. 2026WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMscited by 3
  18. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  19. 2026OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generationcited by 3
  20. 2026EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streamscited by 3
  21. 2025Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AIcited by 2
  22. 2025Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?cited by 2
  23. 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoningcited by 2
  24. 2026M3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Taskscited by 2
  25. 2026LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streamscited by 2
  26. 2026S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrievalcited by 2
  27. 2026ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandingcited by 2
  28. 2025M3-Verse: A "Spot the Difference" Challenge for Large Multimodal Modelscited by 1
  29. 2026Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMUcited by 1
  30. 2026Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasetscited by 1
  31. 2026Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?cited by 1
  32. 2026GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Videocited by 1
  33. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  34. 2025Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Modelscited by 0
  35. 2026Artic: AI-oriented Real-time Communication for MLLM Video Assistantcited by 0
  36. 2026Online Reasoning Video Object Segmentationcited by 0
  37. 2026X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understandingcited by 0
  38. 2026QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understandingcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
  2. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  4. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  5. 2025TextVidBench: A Benchmark for Long Video Scene Text Understanding
  6. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
  7. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
  8. 2024Neptune: The Long Orbit to Benchmarking Long Video Understanding