mlsanityData to 14 Oct 2026

ViSpeak: Visual Instruction Feedback in Streaming Videos

arXiv 2503.12769 ↗First posted 17 Mar 2025cs.CVcited by 42

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Recent advances in Large Multi-modal Models (LMMs) are primarily focused on offline video understanding. Instead, streaming video understanding poses great challenges to recent models due to its time-sensitive, omni-modal and interactive characteristics. In this work, we aim to extend the streaming video understanding from a new perspective and propose a novel task named Visual Instruction Feedback in which models should be aware of visual contents and learn to extract instructions from them. For example, when users wave their hands to agents, agents should recognize the gesture and start conversations with welcome information. Thus, following instructions in visual modality greatly enhances user-agent interactions. To facilitate research, we define seven key subtasks highly relevant to visual modality and collect the ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation. Further, we propose the ViSpeak model, which is a SOTA streaming video understanding LMM with GPT-4o-level performance on various streaming video understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset, ViSpeak is equipped with basic visual instruction feedback ability, serving as a solid baseline for future research.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 77 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

65 of 77 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding15 Feb
  2. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions12 Dec
  3. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding6 Nov
  4. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  5. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities15 Oct
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov

Other references · 57

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
38 more
  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  5. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  8. 2023Instruction Tuning with GPT-4cited by 686
  9. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  11. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  12. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  13. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  14. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  15. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  16. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  17. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  18. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  19. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  20. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  21. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  22. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  23. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  24. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  25. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  26. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  27. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  28. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  29. 2019Oops! Predicting Unintentional Action in Videocited by 82
  30. 2025Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledgecited by 73
  31. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 70
  32. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 70
  33. 2023Grounded Question-Answering in Long Egocentric Videoscited by 64
  34. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  35. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 57
  36. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 47
  37. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  38. 2023FunQA: Towards Surprising Video Comprehensioncited by 35

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 34.

42 papers.

Sort:

By area:

  1. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  2. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 55
  3. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  4. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  5. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  6. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 28
  7. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  8. 2025Streaming Video Instruction Tuningcited by 26
  9. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 22
  10. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  11. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 16
  12. 2025StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videoscited by 8
30 more
  1. 2026Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Modelscited by 8
  2. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 8
  3. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  4. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 7
  5. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 6
  6. 2026MOSS-VL Technical Reportcited by 5
  7. 2026Don't Pause! Every prediction matters in a streaming videocited by 4
  8. 2026StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Videocited by 4
  9. 2026OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistantscited by 4
  10. 2026Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understandingcited by 4
  11. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4
  12. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  13. 2026Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrievalcited by 3
  14. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  15. 2026GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experiencecited by 2
  16. 2026ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandingcited by 2
  17. 2026Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answeringcited by 1
  18. 2026Don't Pause: Streaming Video-Language Synchrony for Online Video Understandingcited by 1
  19. 2026Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?cited by 1
  20. 2026Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understandingcited by 1
  21. 2026SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selectioncited by 1
  22. 2026MEMO: Multi-Level Entity-Aware Memory for Streaming Video Understandingcited by 1
  23. 2026OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interactioncited by 1
  24. 2026GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understandingcited by 0
  25. 2026CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understandingcited by 0
  26. 2026Agentic Video Understanding: A Surveycited by 0
  27. 2026Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learningcited by 0
  28. 2026ReMem: Streaming Video Understanding With Long Context Retentioncited by 0
  29. 2026Proactive Physical AI through Continuous Observation and Timely Interventioncited by 0
  30. 2026DeltaStreamer: Towards Human-Like Agentic Streaming Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding
  2. 2025V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction
  3. 2026VCIFBench: Evaluating Complex Instruction Following for Video Understanding
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning
  5. 2026Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
  6. 2026Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
  7. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts
  8. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding