IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams
On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs
Abstract (arXiv)
Recent multimodal large language models (MLLMs) achieve strong performance on reactive question answering, but real-world streaming assistants require proactive reasoning over continuous visual inputs. Existing benchmarks mainly study reactive or proactive interactions in isolated single-turn settings, overlooking dynamic multi-turn scenarios where users may add, modify, or cancel proactive requests alongside interleaved reactive queries. To address this gap, we introduce IPIBench, the first benchmark for evaluating Interactive Proactive Intelligence of MLLMs under streaming video settings. IPIBench covers proactive monitoring, proactive task management, and interleaved reactive-proactive requests. Evaluations on representative MLLMs reveal two major limitations: unstable proactive triggering and weak coordination between reactive and proactive behaviors. We further propose IPI-Agent, a training-free agentic framework with an interaction-control policy and a temporal-gating mechanism for stabilizing proactive triggering and coordinating multi-turn interactions. Experiments show that IPI-Agent consistently improves existing MLLMs across all benchmark settings.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 58 references that are in the map. First posted on arXiv 26 May 2026.
Builds on
53 of 58 references.
Closest ancestors · 11
Same direction, newest first
- 2026RIVER: A Real-Time Interaction Benchmark for Video LLMs4 Mar
- 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning7 Dec
- 2025Kwai Keye-VL 1.5 Technical Report1 Sep
- 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video4 May
- 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts29 Mar
- 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding15 Feb
- 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling21 Jan
- 2024Online Video Understanding: OVBench and VideoChat-Online31 Dec
- 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding6 Nov
- 2024Long Context Transfer from Language to Vision24 Jun
- 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding13 Jun
Other references · 42
- 2024GPT-4o System Cardcited by 4,588
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
- 2025OpenAI GPT-5 System Cardcited by 989
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
- 2017TALL: Temporal Activity Localization via Language Querycited by 584
30 more
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
- 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
- 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
- 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
- 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
- 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
- 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
- 2026Qwen3.5-Omni Technical Reportcited by 174
- 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
- 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
- 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
- 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
- 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
- 2019Oops! Predicting Unintentional Action in Videocited by 82
- 2025Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledgecited by 73
- 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
- 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
- 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 47
- 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 46
- 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
- 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
- 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 28
- 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 28
- 2023EgoTracks: A Long-term Egocentric Visual Object Tracking Datasetcited by 27
- 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
- 2025Streaming Video Instruction Tuningcited by 26
- 2023Reading Between the Lanes: Text VideoQA on the Roadcited by 23
- 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
- 2025EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wildcited by 7
- 2025StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQAcited by 6
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 6
- 2026SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactionscited by 2
- 2026Learning Multimodal Embeddings with Evidence-Aligned Readoutcited by 1
- 2026From Seeing to Acting: Smart Glasses as First-Person Intelligence Platformscited by 0
- 2026TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understandingcited by 0
- 2026InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systemscited by 0
- 2026Improving Proactive AI Assistance with Hierarchical Procedural Understandingcited by 0
- 2026Improving Proactive AI Assistance with Hierarchical Procedural Understanding5 Oct 2026
- 2026InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems5 Oct 2026
- 2026Learning Multimodal Embeddings with Evidence-Aligned Readout27 Sep 2026
- 2026TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding25 Sep 2026
- 2026SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions30 Aug 2026
- 2026From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms25 Aug 2026
- 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding23 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
- 2026From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench
- 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interaction
- 2026ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response
- 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarios
- 2026Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices
- 2026VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding