mlsanityData to 7 Oct 2026

PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model

arXiv 2410.11564 ↗First posted 15 Oct 2024cs.RO cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Affordance understanding, the task of identifying actionable regions on 3D objects, plays a vital role in allowing robotic systems to engage with and operate within the physical world. Although Visual Language Models (VLMs) have excelled in high-level reasoning and long-horizon planning for robotic manipulation, they still fall short in grasping the nuanced physical properties required for effective human-robot interaction. In this paper, we introduce PAVLM (Point cloud Affordance Vision-Language Model), an innovative framework that utilizes the extensive multimodal knowledge embedded in pre-trained language models to enhance 3D affordance understanding of point cloud. PAVLM integrates a geometric-guided propagation module with hidden embeddings from large language models (LLMs) to enrich visual semantics. On the language side, we prompt Llama-3.1 models to generate refined context-aware text, augmenting the instructional input with deeper semantic cues. Experimental results on the 3D-AffordanceNet benchmark demonstrate that PAVLM outperforms baseline methods for both full and partial point clouds, particularly excelling in its generalization to novel open-world affordance tasks of 3D objects. For more information, visit our project site: pavlm-source.github.io.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 47 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

36 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  12. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
23 more
  1. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  2. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  3. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 237
  4. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  5. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  6. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  7. 2022Human-to-Robot Imitation in the Wildcited by 157
  8. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  9. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  10. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  11. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  12. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  13. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  14. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 47
  15. 2023Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoderscited by 42
  16. 2022PartAfford: Part-level Affordance Discovery from 3D Objectscited by 35
  17. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  18. 2020Affordances in Robotic Tasks -- A Surveycited by 28
  19. 2018Visual Affordance and Function Understanding: A Surveycited by 20
  20. 2018Visual Memory for Robust Path Followingcited by 16
  21. 2023Language-Conditioned Affordance-Pose Detection in 3D Point Cloudscited by 16
  22. 2022Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordancecited by 8
  23. 2024A call for embodied AIcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Modelcited by 2
  2. 2025Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learningcited by 1
  3. 2026HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Groundingcited by 1
  4. 2026CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objectscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
  2. 2024SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model
  3. 2025Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions
  4. 2026StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning
  5. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
  6. 20253D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds
  7. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance
  8. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models