mlsanityData to 7 Oct 2026

Grounding 3D Scene Affordance From Egocentric Interactions

arXiv 2409.19650 ↗First posted 29 Sep 2024cs.CV cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Grounding 3D scene affordance aims to locate interactive regions in 3D environments, which is crucial for embodied agents to interact intelligently with their surroundings. Most existing approaches achieve this by mapping semantics to 3D instances based on static geometric structure and visual appearance. This passive strategy limits the agent's ability to actively perceive and engage with the environment, making it reliant on predefined semantic instructions. In contrast, humans develop complex interaction skills by observing and imitating how others interact with their surroundings. To empower the model with such abilities, we introduce a novel task: grounding 3D scene affordance from egocentric interactions, where the goal is to identify the corresponding affordance regions in a 3D scene based on an egocentric video of an interaction. This task faces the challenges of spatial complexity and alignment complexity across multiple sources. To address these challenges, we propose the Egocentric Interaction-driven 3D Scene Affordance Grounding (Ego-SAG) framework, which utilizes interaction intent to guide the model in focusing on interaction-relevant sub-regions and aligns affordance features from different sources through a bidirectional query decoder mechanism. Furthermore, we introduce the Egocentric Video-3D Scene Affordance Dataset (VSAD), covering a wide range of common interaction types and diverse 3D environments to support this task. Extensive experiments on VSAD validate both the feasibility of the proposed task and the effectiveness of our approach.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

51 of 70 references.

References · 51

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2020Rescaling Egocentric Visioncited by 529
  12. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
38 more
  1. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  2. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  3. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  4. 2022Learning Video Representations from Large Language Modelscited by 204
  5. 2016What do different evaluation metrics tell us about saliency models?cited by 184
  6. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  7. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  8. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  9. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  10. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  11. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  12. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  13. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  14. 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
  15. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  16. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  17. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  18. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  19. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  20. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  21. 2020Learning Affordance Landscapes for Interaction Exploration in 3D Environmentscited by 65
  22. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  23. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  24. 2021Forecasting Action through Contact Representations from First Person Videocited by 46
  25. 2023Mask-Attention-Free Transformer for 3D Instance Segmentationcited by 45
  26. 2021One-Shot Object Affordance Detection in the Wildcited by 40
  27. 2023Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perceptioncited by 40
  28. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  29. 2022PartAfford: Part-level Affordance Discovery from 3D Objectscited by 35
  30. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  31. 2021One-Shot Affordance Detectioncited by 32
  32. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  33. 2023Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learningcited by 26
  34. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  35. 2023StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipationcited by 23
  36. 2022Phrase-Based Affordance Detection via Cyclic Bilateral Interactioncited by 19
  37. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  38. 2021Affordance-Based Mobile Robot Navigation Among Movable Obstaclescited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 9.

11 papers.

  1. 2024GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Groundingcited by 26
  2. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  3. 2026Pandora: Articulated 3D Scene Graphs from Egocentric Visioncited by 3
  4. 2026Gloria: Consistent Character Video Generation via Content Anchorscited by 3
  5. 2026CoT-Edit: Let CoT Guide Instruction Video Editingcited by 3
  6. 2026VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Modelcited by 2
  7. 2026VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videoscited by 1
  8. 2026CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objectscited by 1
  9. 2026EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templatescited by 1
  10. 2026AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifierscited by 0
  11. 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0