# Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video

arXiv: https://arxiv.org/abs/2510.14560 · HTML: https://mlsanity.com/paper/2510.14560/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding](https://mlsanity.com/paper/2409.14485/) (2024)
- [Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2406.08085/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://mlsanity.com/paper/2310.16436/) (2023)
- [Streaming Video Question-Answering with In-context Video KV-Cache Retrieval](https://mlsanity.com/paper/2503.00540/) (2025)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning](https://mlsanity.com/paper/2312.06722/) (2023)
- [VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation](https://mlsanity.com/paper/2408.16730/) (2024)
- [Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding](https://mlsanity.com/paper/2503.18478/) (2025)
- [VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format](https://mlsanity.com/paper/2411.17991/) (2024)
- [Efficient Large Multi-modal Models via Visual Context Compression](https://mlsanity.com/paper/2406.20092/) (2024)
- [EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios](https://mlsanity.com/paper/2412.04447/) (2024)
- [xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs](https://mlsanity.com/paper/2410.16267/) (2024)

## Built on by (most cited first, 28 of 28)

- [A Simple Baseline for Streaming Video Understanding](https://mlsanity.com/paper/2604.02317/) (2026)
- [Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2603.11896/) (2026)
- [Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding](https://mlsanity.com/paper/2603.19054/) (2026)
- [IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams](https://mlsanity.com/paper/2605.27074/) (2026)
- [EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding](https://mlsanity.com/paper/2606.24422/) (2026)
- [Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos](https://mlsanity.com/paper/2607.11523/) (2026)
- [STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding](https://mlsanity.com/paper/2603.27593/) (2026)
- [StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video](https://mlsanity.com/paper/2605.16381/) (2026)
- [OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants](https://mlsanity.com/paper/2605.26485/) (2026)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [FOLIO: Focused Semantic Memory for Streaming Video Understanding](https://mlsanity.com/paper/2607.13298/) (2026)
- [WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs](https://mlsanity.com/paper/2602.22142/) (2026)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams](https://mlsanity.com/paper/2605.07299/) (2026)
- [RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generation](https://mlsanity.com/paper/2601.22094/) (2026)
- [Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding](https://mlsanity.com/paper/2605.07575/) (2026)
- [GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience](https://mlsanity.com/paper/2608.02392/) (2026)
- [Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation](https://mlsanity.com/paper/2609.03727/) (2026)
- [EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning](https://mlsanity.com/paper/2605.17262/) (2026)
- [Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding](https://mlsanity.com/paper/2606.06991/) (2026)
- [X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding](https://mlsanity.com/paper/2606.02482/) (2026)
- [EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports](https://mlsanity.com/paper/2608.25561/) (2026)
- [TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding](https://mlsanity.com/paper/2609.30670/) (2026)
- [Agentic Video Understanding: A Survey](https://mlsanity.com/paper/2609.31713/) (2026)
- [APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants](https://mlsanity.com/paper/2609.37559/) (2026)
- [Foresight: planning future perception in streaming VLMs without retraining](https://mlsanity.com/paper/2610.03123/) (2026)
- [InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems](https://mlsanity.com/paper/2610.05775/) (2026)
- [Proactive Physical AI through Continuous Observation and Timely Intervention](https://mlsanity.com/paper/2610.16967/) (2026)

## Abstract (arXiv)

Envision an AI capable of functioning in human-like settings, moving beyond mere observation to actively understand, anticipate, and proactively respond to unfolding events. Towards this vision, we focus on the innovative task where, given ego-streaming video input, an assistant proactively answers diverse, evolving questions at the opportune moment, while maintaining synchronized perception and reasoning. This task embodies three key properties: (1) Proactive Coherence, (2) Just-in-Time Responsiveness, and (3) Synchronized Efficiency. To evaluate and address these properties, we first introduce ESTP-Bench (Ego Streaming Proactive Benchmark) alongside the ESTP-F1 metric-a novel framework designed for their rigorous assessment. Secondly, we propose a comprehensive technical pipeline to enable models to tackle this challenging task. This pipeline comprises: (1) a data engine, (2) a multi-stage training strategy, and (3) a proactive dynamic compression technique. Our proposed model effectively addresses these critical properties while outperforming multiple baselines across diverse online and offline benchmarks. Project Page:https://zhangyl4.github.io/publications/eyes-wide-open/