mlsanityData to 7 Oct 2026

Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model

arXiv 2412.21080 ↗First posted 30 Dec 2024cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-language model. Designed for deployment on portable devices such as smartphones and wearable cameras, Vinci operates in an "always on" mode, continuously observing the environment to deliver seamless interaction and assistance. Users can wake up the system and engage in natural conversations to ask questions or seek assistance, with responses delivered through audio for hands-free convenience. With its ability to process long video streams in real-time, Vinci can answer user queries about current observations and historical context while also providing task planning based on past interactions. To further enhance usability, Vinci integrates a video generation module that creates step-by-step visual demonstrations for tasks that require detailed guidance. We hope that Vinci can establish a robust framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. We release the complete implementation for the development of the device in conjunction with a demo web platform to test uploaded videos at https://github.com/OpenGVLab/vinci.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 91 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

67 of 91 references.

References · 67

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  11. 2020Rescaling Egocentric Visioncited by 529
  12. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
38 more
  1. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  2. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  3. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  4. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  5. 2022Egocentric Video-Language Pretrainingcited by 231
  6. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  7. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  8. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  9. 2022Learning Video Representations from Large Language Modelscited by 204
  10. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  11. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  12. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  13. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  14. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  15. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  16. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  17. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  18. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  19. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  20. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
  21. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  22. 2023An Outlook into the Future of Egocentric Visioncited by 84
  23. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  24. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  25. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  26. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  27. 2020Learning Affordance Landscapes for Interaction Exploration in 3D Environmentscited by 65
  28. 2018Jointly Localizing and Describing Events for Dense Video Captioningcited by 60
  29. 2019Mutual Context Network for Jointly Estimating Egocentric Gaze and Actionscited by 54
  30. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 49
  31. 2022InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challengescited by 48
  32. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 46
  33. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  34. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  35. 2020Integrating Human Gaze into Attention for Egocentric Activity Recognitioncited by 42
  36. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  37. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  38. 2021DCAN: Improving Temporal Action Detection via Dual Context Aggregationcited by 36

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  2. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  3. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  4. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  5. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  6. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  7. 2025X-LeBench: A Benchmark for Extremely Long Egocentric Video Understandingcited by 14
  8. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  9. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  10. 2025MICA: Multi-Agent Industrial Coordination Assistantcited by 8
  11. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  12. 2025Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidancecited by 2
7 more
  1. 2026STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understandingcited by 2
  2. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 2
  3. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 2
  4. 2026Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistantcited by 1
  5. 2025Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novelscited by 0
  6. 2026GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearablescited by 0
  7. 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025An Egocentric Vision-Language Model based Portable Real-time Smart Assistant
  2. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  3. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
  4. 2024AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding
  5. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
  6. 2025VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting
  7. 2026LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks
  8. 2026Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI