Affordance Perception by a Knowledge-Guided Vision-Language Model with Efficient Error Correction
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Mobile robot platforms will increasingly be tasked with activities that involve grasping and manipulating objects in open world environments. Affordance understanding provides a robot with means to realise its goals and execute its tasks, e.g. to achieve autonomous navigation in unknown buildings where it has to find doors and ways to open these. In order to get actionable suggestions, robots need to be able to distinguish subtle differences between objects, as they may result in different action sequences: doorknobs require grasp and twist, while handlebars require grasp and push. In this paper, we improve affordance perception for a robot in an open-world setting. Our contribution is threefold: (1) We provide an affordance representation with precise, actionable affordances; (2) We connect this knowledge base to a foundational vision-language models (VLM) and prompt the VLM for a wider variety of new and unseen objects; (3) We apply a human-in-the-loop for corrections on the output of the VLM. The mix of affordance representation, image detection and a human-in-the-loop is effective for a robot to search for objects to achieve its goals. We have demonstrated this in a scenario of finding various doors and the many different ways to open them.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 37 references that are in the map. First posted on arXiv 18 Jul 2024.
Builds on
16 of 37 references.
References · 16
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2021Grounded Language-Image Pre-trainingcited by 1,147
- 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
- 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
- 2024From human experts to machines: An LLM supported approach to ontology and knowledge graph constructioncited by 40
- 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
- 2020Affordances in Robotic Tasks -- A Surveycited by 28
- 2022TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillationcited by 25
- 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
- 2019What Object Should I Use? - Task Driven Object Detectioncited by 22
4 more
- 2022NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledgecited by 10
- 2021Building Affordance Relations for Robotic Agents - A Reviewcited by 7
- 2020Affordance as general value function: A computational modelcited by 5
- 2022A General Framework for the Representation of Function and Affordance: A Cognitive, Causal, and Grounded Approach, and a Step Toward AGIcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Which objects help me to act effectively? Reasoning about physically-grounded affordances
- 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance
- 2025AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
- 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
- 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
- 2024KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data
- 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping
- 2024PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model