# ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

arXiv: https://arxiv.org/abs/2507.09313 · HTML: https://mlsanity.com/paper/2507.09313/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 12 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension](https://mlsanity.com/paper/2404.16790/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025)
- [Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction](https://mlsanity.com/paper/2501.03218/) (2025)
- [OmniBench: Towards The Future of Universal Omni-Language Models](https://mlsanity.com/paper/2409.15272/) (2024)
- [TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models](https://mlsanity.com/paper/2410.10818/) (2024)
- [TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos](https://mlsanity.com/paper/2504.17343/) (2025)
- [E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding](https://mlsanity.com/paper/2409.18111/) (2024)
- [VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models](https://mlsanity.com/paper/2311.17404/) (2023)
- [Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos](https://mlsanity.com/paper/2312.10300/) (2023)
- [OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts](https://mlsanity.com/paper/2503.22952/) (2025)
- [VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format](https://mlsanity.com/paper/2411.17991/) (2024)
- [Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges](https://mlsanity.com/paper/2309.13925/) (2023)

## Built on by (most cited first, 27 of 27)

- [A Simple Baseline for Streaming Video Understanding](https://mlsanity.com/paper/2604.02317/) (2026)
- [MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2512.06810/) (2025)
- [StreamReady: Learning What to Answer and When in Long Streaming Videos](https://mlsanity.com/paper/2603.08620/) (2026)
- [Proact-VL: A Proactive VideoLLM for Real-Time AI Companions](https://mlsanity.com/paper/2603.03447/) (2026)
- [Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents](https://mlsanity.com/paper/2510.19771/) (2025)
- [PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios](https://mlsanity.com/paper/2601.22575/) (2026)
- [VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding](https://mlsanity.com/paper/2607.14935/) (2026)
- [Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding](https://mlsanity.com/paper/2603.19054/) (2026)
- [IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams](https://mlsanity.com/paper/2605.27074/) (2026)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models](https://mlsanity.com/paper/2602.02039/) (2026)
- [MOSS-VL Technical Report](https://mlsanity.com/paper/2608.15045/) (2026)
- [From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench](https://mlsanity.com/paper/2604.15037/) (2026)
- [Don't Pause! Every prediction matters in a streaming video](https://mlsanity.com/paper/2604.24317/) (2026)
- [StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video](https://mlsanity.com/paper/2605.16381/) (2026)
- [Realtime-Venus: A full-duplex interaction system with asynchronous delegation](https://mlsanity.com/paper/2609.13814/) (2026)
- [Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning](https://mlsanity.com/paper/2605.28192/) (2026)
- [Qwen3.8-Omni: Towards Native Omni-Modal Agents](https://mlsanity.com/paper/2609.25611/) (2026)
- [GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video](https://mlsanity.com/paper/2607.02991/) (2026)
- [Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning](https://mlsanity.com/paper/2608.15869/) (2026)
- [OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction](https://mlsanity.com/paper/2610.01762/) (2026)
- [X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding](https://mlsanity.com/paper/2606.02482/) (2026)
- [OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs](https://mlsanity.com/paper/2608.21360/) (2026)
- [TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding](https://mlsanity.com/paper/2609.30670/) (2026)
- [LLMs are General Asynchronous Agents](https://mlsanity.com/paper/2609.35427/) (2026)
- [Improving Proactive AI Assistance with Hierarchical Procedural Understanding](https://mlsanity.com/paper/2610.06505/) (2026)
- [Proactive Physical AI through Continuous Observation and Timely Intervention](https://mlsanity.com/paper/2610.16967/) (2026)

## Abstract (arXiv)

With the growing research focus on multimodal dialogue systems, the capability for proactive interaction is gradually gaining recognition. As an alternative to conventional turn-by-turn dialogue, users increasingly expect multimodal systems to be more initiative, for example, by autonomously determining the timing of multi-turn responses in real time during video playback. To facilitate progress in this emerging area, we introduce ProactiveVideoQA, the first comprehensive benchmark to evaluate a system's ability to engage in proactive interaction. Since model responses are generated at varying timestamps, we further propose PAUC, the first metric that accounts for the temporal dynamics of model responses. This enables a more accurate evaluation of systems operating in proactive settings. Through extensive benchmarking of various baseline systems on ProactiveVideoQA and a user study of human preferences, we show that PAUC is in better agreement with human preferences than traditional evaluation metrics, which typically only consider the textual content of responses. These findings demonstrate that PAUC provides a more faithful assessment of user experience in proactive interaction scenarios. Project homepage: https://github.com/yellow-binary-tree/ProactiveVideoQA