mlsanityData to 7 Oct 2026

AffordanceLLM: Grounding Affordance from Vision Language Models

arXiv 2401.06341 ↗First posted 12 Jan 2024cs.CV cs.ROcited by 55

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Affordance grounding refers to the task of finding the area of an object with which one can interact. It is a fundamental but challenging task, as a successful solution requires the comprehensive understanding of a scene in multiple aspects including detection, localization, and recognition of objects with their parts, of geo-spatial configuration/layout of the scene, of 3D shapes and physics, as well as of the functionality and potential interaction of the objects and humans. Much of the knowledge is hidden and beyond the image content with the supervised labels from a limited training set. In this paper, we make an attempt to improve the generalization capability of the current affordance grounding by taking the advantage of the rich world, abstract, and human-object-interaction knowledge from pretrained large-scale vision language models. Under the AGD20K benchmark, our proposed model demonstrates a significant performance gain over the competing methods for in-the-wild object affordance grounding. We further demonstrate it can ground affordance for objects from random Internet images, even if both objects and actions are unseen during training. Project site: https://jasonqsy.github.io/AffordanceLLM/

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 76 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

65 of 76 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2021Vision Transformers for Dense Predictioncited by 1,789
38 more
  1. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  2. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  3. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2022Language-driven Semantic Segmentationcited by 711
  7. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2021Image Segmentation Using Text and Image Promptscited by 509
  10. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  11. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  12. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  13. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  14. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  15. 2019Enforcing geometric constraints of virtual normal for depth predictioncited by 233
  16. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  17. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  18. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  19. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
  20. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  21. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  22. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  23. 2022Human-to-Robot Imitation in the Wildcited by 157
  24. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  25. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  26. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  27. 2018PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Imagecited by 105
  28. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  29. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  30. 2019Does computer vision matter for action?cited by 64
  31. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  32. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  33. 2014Designing Deep Networks for Surface Normal Estimationcited by 59
  34. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  35. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 47
  36. 2023CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detectioncited by 46
  37. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignmentcited by 44
  38. 2023Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perceptioncited by 40

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 25; 2026: 18.

55 papers.

Sort:

By area:

  1. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  2. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  3. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  4. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  5. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  6. 2024GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Groundingcited by 26
  7. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Modelscited by 25
  8. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  9. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  10. 2025Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactionscited by 19
  11. 2025OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Modelcited by 17
  12. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
38 more
  1. 2024Multi-GraspLLM: A Multimodal LLM for Multi-Hand Semantic Guided Grasp Generationcited by 12
  2. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  3. 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11
  4. 20252HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videoscited by 9
  5. 2025Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priorscited by 9
  6. 2025AffordanceSAM: Segment Anything Once More in Affordance Groundingcited by 6
  7. 2025Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scalecited by 6
  8. 2025Selective Contrastive Learning for Weakly Supervised Affordance Groundingcited by 6
  9. 2024Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraintscited by 5
  10. 2025Physically Ground Commonsense Knowledge for Articulated Object Manipulation with Analytic Conceptscited by 5
  11. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignmentcited by 5
  12. 2024PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Modelcited by 4
  13. 2025Closed-Loop Transfer for Weakly-supervised Affordance Groundingcited by 4
  14. 2025Advances in 4D Representation: Geometry, Motion, and Interactioncited by 4
  15. 2025Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulationcited by 3
  16. 2025O3Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulationcited by 3
  17. 2025FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulationcited by 3
  18. 2024SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interactioncited by 2
  19. 2025GraspCoT: Integrating Physical Property Reasoning for 6-DoF Grasping under Flexible Language Instructionscited by 2
  20. 2025Accessibility Scout: Personalized Accessibility Scans of Built Environmentscited by 2
  21. 2025Weakly-Supervised Learning of Dense Functional Correspondencescited by 2
  22. 2025Cracking the Code of Action: a Generative Approach to Affordances for Reinforcement Learningcited by 1
  23. 2025Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learningcited by 1
  24. 2025Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Actioncited by 1
  25. 2026Affostruction: 3D Affordance Grounding with Generative Reconstructioncited by 1
  26. 2026AffordanceGrasp-R1:Leveraging Reasoning-Based Affordance Segmentation with Reinforcement Learning for Robotic Graspingcited by 1
  27. 2026Panoramic Affordance Predictioncited by 1
  28. 2026CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objectscited by 1
  29. 2025Capturing Fine-Grained Alignments Improves 3D Affordance Detectioncited by 0
  30. 2025Learning Visual Affordance from Audiocited by 0
  31. 2025YOLOA: Real-Time Affordance Detection via LLM Adaptercited by 0
  32. 2026Probing and Bridging Geometry-Interaction Cues for Affordance Reasoning in Vision Foundation Modelscited by 0
  33. 2026Context-Dependent Affordance Reports in Vision-Language Modelscited by 0
  34. 2026UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentationcited by 0
  35. 2026SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenescited by 0
  36. 2026SECOND-Grasp: Semantic Contact-guided Dexterous Graspingcited by 0
  37. 2026SceneParser: Hierarchical Scene Parsing for Visual Semantics Understandingcited by 0
  38. 2026GROW2: Grounding Which and Where for Robot Tool Usecited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024WorldAfford: Affordance Grounding based on Natural Language Instructions
  2. 2024INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding
  3. 2026AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning
  4. 2024TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Text
  5. 2026Token-Based Affordance Grounding with Large Vision-Language Models
  6. 2026SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance
  7. 2026Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
  8. 2026VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Model