mlsanityData to 14 Oct 2026

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

arXiv 2608.18671 ↗First posted 19 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior. This perspective is increasingly important for wearable intelligence, assistive systems, human--robot interaction, and embodied AI, yet it introduces challenges including ego-motion, occlusion, small active objects, viewpoint-dependent appearance, and long-range temporal dependencies. Vision--language models (VLMs) offer a promising foundation for addressing these challenges by linking visual observations with semantic knowledge and natural-language supervision. This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems. We organize the literature around tasks, datasets, hand--object interaction understanding, temporal reasoning, frame and clip selection, multimodal representation learning, prompting, semantic alignment, and model adaptation. Particular attention is given to graph-based and object-centric reasoning as mechanisms for modeling relations among hands, objects, actions, and scene context over time. We further examine how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making. Across the reviewed literature, a consistent limitation emerges: current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities. We therefore identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 0 of its 165 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

0 of 165 references.

No references here yet.

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
  2. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision
  3. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
  4. 2025Developing Vision-Language-Action Model from Egocentric Videos
  5. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
  6. 2024EgoLM: Multi-Modal Language Model of Egocentric Motions
  7. 2024AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding
  8. 2026EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting