mlsanityData to 14 Oct 2026

Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video

arXiv 2510.14560 ↗First posted 16 Oct 2025cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

Envision an AI capable of functioning in human-like settings, moving beyond mere observation to actively understand, anticipate, and proactively respond to unfolding events. Towards this vision, we focus on the innovative task where, given ego-streaming video input, an assistant proactively answers diverse, evolving questions at the opportune moment, while maintaining synchronized perception and reasoning. This task embodies three key properties: (1) Proactive Coherence, (2) Just-in-Time Responsiveness, and (3) Synchronized Efficiency. To evaluate and address these properties, we first introduce ESTP-Bench (Ego Streaming Proactive Benchmark) alongside the ESTP-F1 metric-a novel framework designed for their rigorous assessment. Secondly, we propose a comprehensive technical pipeline to enable models to tackle this challenging task. This pipeline comprises: (1) a data engine, (2) a multi-stage training strategy, and (3) a proactive dynamic compression technique. Our proposed model effectively addresses these critical properties while outperforming multiple baselines across diverse online and offline benchmarks. Project Page:https://zhangyl4.github.io/publications/eyes-wide-open/

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

34 of 56 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  11. 2023Large Language Modelscited by 780
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
22 more
  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  2. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  3. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  4. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  5. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  6. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  7. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  8. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  9. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  10. 2024Long Context Compression with Activation Beaconcited by 104
  11. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 99
  12. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 69
  13. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  14. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  15. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 47
  16. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  17. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 41
  18. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  19. 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QAcited by 24
  20. 2024InfiniBench: A Benchmark for Large Multi-Modal Models in Long-Form Movies and TV Showscited by 20
  21. 2024Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memorycited by 11
  22. 2024Streaming Detection of Queried Event Startcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 28.

28 papers.

Sort:

By area:

  1. 2026A Simple Baseline for Streaming Video Understandingcited by 33
  2. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 8
  3. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 7
  4. 2026IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streamscited by 7
  5. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 6
  6. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 5
  7. 2026STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understandingcited by 4
  8. 2026StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Videocited by 4
  9. 2026OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistantscited by 4
  10. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  11. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4
  12. 2026WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMscited by 3
16 more
  1. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  2. 2026EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streamscited by 3
  3. 2026RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generationcited by 2
  4. 2026Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understandingcited by 2
  5. 2026GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experiencecited by 2
  6. 2026Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluationcited by 2
  7. 2026EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoningcited by 1
  8. 2026Don't Pause: Streaming Video-Language Synchrony for Online Video Understandingcited by 1
  9. 2026X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understandingcited by 0
  10. 2026EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supportscited by 0
  11. 2026TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understandingcited by 0
  12. 2026Agentic Video Understanding: A Surveycited by 0
  13. 2026APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistantscited by 0
  14. 2026Foresight: planning future perception in streaming VLMs without retrainingcited by 0
  15. 2026InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systemscited by 0
  16. 2026Proactive Physical AI through Continuous Observation and Timely Interventioncited by 0