mlsanityData to 7 Oct 2026

Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions

arXiv 2504.04744 ↗First posted 7 Apr 2025cs.CV cs.AI cs.ROcited by 19

Not on the map. Why?

Abstract (arXiv)

Grounding 3D object affordance is a task that locates objects in 3D space where they can be manipulated, which links perception and action for embodied intelligence. For example, for an intelligent robot, it is necessary to accurately ground the affordance of an object and grasp it according to human instructions. In this paper, we introduce a novel task that grounds 3D object affordance based on language instructions, visual observations and interactions, which is inspired by cognitive science. We collect an Affordance Grounding dataset with Points, Images and Language instructions (AGPIL) to support the proposed task. In the 3D physical world, due to observation orientation, object rotation, or spatial occlusion, we can only get a partial observation of the object. So this dataset includes affordance estimations of objects from full-view, partial-view, and rotation-view perspectives. To accomplish this task, we propose LMAffordance3D, the first multi-modal, language-guided 3D affordance grounding network, which applies a vision-language model to fuse 2D and 3D spatial features with semantic features. Comprehensive experiments on AGPIL demonstrate the effectiveness and superiority of our method on this task, even in unseen experimental settings. Our project is available at https://sites.google.com/view/lmaffordance3d.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

34 of 57 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
22 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  7. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 135
  8. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  9. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  10. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  11. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  12. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  13. 2020Learning Affordance Landscapes for Interaction Exploration in 3D Environmentscited by 65
  14. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  15. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  16. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  17. 2023Open-Vocabulary Affordance Detection in 3D Point Cloudscited by 41
  18. 2022PartAfford: Part-level Affordance Discovery from 3D Objectscited by 35
  19. 2021One-Shot Affordance Detectioncited by 32
  20. 2022PointConvFormer: Revenge of the Point-based Convolutioncited by 29
  21. 2023Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusionscited by 23
  22. 2022Grounded Affordance from Exocentric Viewcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  2. 2025Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scalecited by 6
  3. 2025Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learningcited by 5
  4. 2025FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulationcited by 3
  5. 2025Unlocking 3D Affordance Segmentation with 2D Semantic Knowledgecited by 2
  6. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  7. 2026VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Modelcited by 2
  8. 2026Affostruction: 3D Affordance Grounding with Generative Reconstructioncited by 1
  9. 2026VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videoscited by 1
  10. 2026CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objectscited by 1
  11. 2025YOLOA: Real-Time Affordance Detection via LLM Adaptercited by 0
  12. 2026Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignmentcited by 0
7 more
  1. 2026BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fieldscited by 0
  2. 2026VoxAfford: Multi-Scale Voxel-Token Fusion for Open-Vocabulary 3D Affordance Detectioncited by 0
  3. 2026SECOND-Grasp: Semantic Contact-guided Dexterous Graspingcited by 0
  4. 2026AFUN: Towards an Affordance Foundation Model for Functionality Understandingcited by 0
  5. 2026PruneGround: Plug-and-play Spatial Pruning for 3D Visual Groundingcited by 0
  6. 2026Zero-shot 2D Grounding with Novel Affordance Typescited by 0
  7. 2026UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perceptioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024WorldAfford: Affordance Grounding based on Natural Language Instructions
  2. 2026AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning
  3. 2026Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
  4. 2026ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes
  5. 2026HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding
  6. 2024PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model
  7. 2024SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model
  8. 2026SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance