mlsanityData to 7 Oct 2026

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

arXiv 2607.11523 ↗First posted 13 Jul 2026cs.CV cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at hrefhttps://sitonggong.github.io/EgoServe-page/Vinci2.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 63 of its 77 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

63 of 77 references.

References · 63

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  11. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
38 more
  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  3. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  4. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  5. 2022Egocentric Video-Language Pretrainingcited by 231
  6. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  7. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  8. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  9. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  10. 2022Learning Video Representations from Large Language Modelscited by 204
  11. 2021Anticipative Video Transformercited by 197
  12. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  13. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  14. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  15. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  16. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  17. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  18. 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 112
  19. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  20. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  21. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  22. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  23. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  24. 2023An Outlook into the Future of Egocentric Visioncited by 84
  25. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  26. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  27. 2023A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospectscited by 66
  28. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
  29. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  30. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  31. 2019Mutual Context Network for Jointly Estimating Egocentric Gaze and Actionscited by 54
  32. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  33. 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
  34. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  35. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  36. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  37. 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 44
  38. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision
  2. 2026EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams
  3. 2025An Egocentric Vision-Language Model based Portable Real-time Smart Assistant
  4. 2026EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
  5. 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
  6. 2025In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
  7. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding
  8. 2026EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next