# IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams

arXiv: https://arxiv.org/abs/2605.27074 · HTML: https://mlsanity.com/paper/2605.27074/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 53 of its 58 references that are in the map.
- First posted on arXiv 26 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 53)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2406.08085/) (2024)
- [Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution](https://mlsanity.com/paper/2409.12961/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025)
- [Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction](https://mlsanity.com/paper/2501.03218/) (2025)
- [VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding](https://mlsanity.com/paper/2406.09418/) (2024)
- [TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models](https://mlsanity.com/paper/2410.10818/) (2024)
- [Oops! Predicting Unintentional Action in Video](https://mlsanity.com/paper/1911.11206/) (2019)

## Built on by (most cited first, 7 of 7)

- [EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding](https://mlsanity.com/paper/2606.24422/) (2026)
- [SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions](https://mlsanity.com/paper/2608.29607/) (2026)
- [Learning Multimodal Embeddings with Evidence-Aligned Readout](https://mlsanity.com/paper/2609.33659/) (2026)
- [From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms](https://mlsanity.com/paper/2608.24877/) (2026)
- [TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding](https://mlsanity.com/paper/2609.30670/) (2026)
- [InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems](https://mlsanity.com/paper/2610.05775/) (2026)
- [Improving Proactive AI Assistance with Hierarchical Procedural Understanding](https://mlsanity.com/paper/2610.06505/) (2026)

## Abstract (arXiv)

Recent multimodal large language models (MLLMs) achieve strong performance on reactive question answering, but real-world streaming assistants require proactive reasoning over continuous visual inputs. Existing benchmarks mainly study reactive or proactive interactions in isolated single-turn settings, overlooking dynamic multi-turn scenarios where users may add, modify, or cancel proactive requests alongside interleaved reactive queries. To address this gap, we introduce IPIBench, the first benchmark for evaluating Interactive Proactive Intelligence of MLLMs under streaming video settings. IPIBench covers proactive monitoring, proactive task management, and interleaved reactive-proactive requests. Evaluations on representative MLLMs reveal two major limitations: unstable proactive triggering and weak coordination between reactive and proactive behaviors. We further propose IPI-Agent, a training-free agentic framework with an interaction-control policy and a temporal-gating mechanism for stabilizing proactive triggering and coordinating multi-turn interactions. Experiments show that IPI-Agent consistently improves existing MLLMs across all benchmark settings.