Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
Not on the map. Why?
Abstract (arXiv)
When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at hrefhttps://sitonggong.github.io/EgoServe-page/Vinci2.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 63 of its 77 references that are in the map. First posted on arXiv 13 Jul 2026.
Builds on
63 of 77 references.
References · 63
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Qwen Technical Reportcited by 3,273
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2023VideoChat: Chat-Centric Video Understandingcited by 851
- 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
- 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
38 more
- 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
- 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
- 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
- 2020Understanding Human Hands in Contact at Internet Scalecited by 281
- 2022Egocentric Video-Language Pretrainingcited by 231
- 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
- 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
- 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
- 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
- 2022Learning Video Representations from Large Language Modelscited by 204
- 2021Anticipative Video Transformercited by 197
- 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
- 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
- 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
- 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
- 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
- 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 112
- 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
- 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
- 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
- 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
- 2025EgoLife: Towards Egocentric Life Assistantcited by 86
- 2023An Outlook into the Future of Egocentric Visioncited by 84
- 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
- 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
- 2023A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospectscited by 66
- 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
- 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
- 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
- 2019Mutual Context Network for Jointly Estimating Egocentric Gaze and Actionscited by 54
- 2021Human Hands as Probes for Interactive Object Understandingcited by 51
- 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
- 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
- 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
- 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
- 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 44
- 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision
- 2026EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams
- 2025An Egocentric Vision-Language Model based Portable Real-time Smart Assistant
- 2026EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
- 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
- 2025In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
- 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding
- 2026EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next