mlsanityData to 14 Oct 2026

Simultaneous Localization and Affordance Prediction of Tasks from Egocentric Video

arXiv 2407.13856 ↗First posted 18 Jul 2024cs.RO cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) have shown great success as foundational models for downstream vision and natural language applications in a variety of domains. However, these models are limited to reasoning over objects and actions currently visible on the image plane. We present a spatial extension to the VLM, which leverages spatially-localized egocentric video demonstrations to augment VLMs in two ways -- through understanding spatial task-affordances, i.e. where an agent must be for the task to physically take place, and the localization of that task relative to the egocentric viewer. We show our approach outperforms the baseline of using a VLM to map similarity of a task's description over a set of location-tagged images. Our approach has less error both on predicting where a task may take place and on predicting what tasks are likely to happen at the current location. The resulting representation will enable robots to use egocentric sensing to navigate to, or around, physical regions of interest for novel tasks specified in natural language.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 46 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

35 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  6. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  8. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  9. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  11. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  12. 2023Scaling Open-Vocabulary Object Detectioncited by 343
22 more
  1. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  2. 2022Egocentric Video-Language Pretrainingcited by 232
  3. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  4. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 208
  5. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  6. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  7. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  8. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  9. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  10. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 128
  11. 2022Navigating to Objects in the Real Worldcited by 106
  12. 2023An Outlook into the Future of Egocentric Visioncited by 84
  13. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  14. 2021CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulationcited by 49
  15. 2021Learning a State Representation and Navigation in Cluttered and Dynamic Environmentscited by 41
  16. 2022Is Mapping Necessary for Realistic PointGoal Navigation?cited by 35
  17. 2022Scene-aware Egocentric 3D Human Pose Estimationcited by 30
  18. 2020Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agentscited by 27
  19. 2021Shaping embodied agent behavior with activity-context priors from egocentric videocited by 17
  20. 2021Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusioncited by 17
  21. 2017Reuleaux: Robot Base Placement by Reachability Analysiscited by 11
  22. 2024Text-driven Affordance Learning from Egocentric Visioncited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understanding
  2. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  3. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
  4. 2026EgoAfford: Task-Oriented Affordance Grounding via Egocentric Referring Segmentation
  5. 2026Spatial-Conditioned Reasoning in Long-Egocentric Videos
  6. 2026From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
  7. 2026EgoActor: Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-Language Models
  8. 2025Developing Vision-Language-Action Model from Egocentric Videos