# CRAFT-E: A Neuro-Symbolic Framework for Embodied Affordance Grounding

arXiv: https://arxiv.org/abs/2512.04231 · HTML: https://mlsanity.com/paper/2512.04231/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 3 Dec 2025.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [Shape Completion Enabled Robotic Grasping](https://mlsanity.com/paper/1609.08546/) (2016)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection](https://mlsanity.com/paper/2309.01093/) (2023)
- [Robot Object Retrieval with Contextual Natural Language Queries](https://mlsanity.com/paper/2006.13253/) (2020)
- [UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation](https://mlsanity.com/paper/2506.09284/) (2025)
- [What Object Should I Use? - Task Driven Object Detection](https://mlsanity.com/paper/1904.03000/) (2019)
- [TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection](https://mlsanity.com/paper/2403.08108/) (2024)
- [Open-Vocabulary Affordance Detection using Knowledge Distillation and Text-Point Correlation](https://mlsanity.com/paper/2309.10932/) (2023)
- [What does CLIP know about peeling a banana?](https://mlsanity.com/paper/2404.12015/) (2024)
- [GKNet: grasp keypoint network for grasp candidates detection](https://mlsanity.com/paper/2106.08497/) (2021)
- [Discovering Novel Actions from Open World Egocentric Videos with Object-Grounded Visual Commonsense Reasoning](https://mlsanity.com/paper/2305.16602/) (2023)
- [PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?](https://mlsanity.com/paper/2506.23725/) (2025)
- [CRAFT: A Neuro-Symbolic Framework for Visual Functional Affordance Grounding](https://mlsanity.com/paper/2507.14426/) (2025)

## Abstract (arXiv)

Assistive robots operating in unstructured environments must understand not only what objects are, but what they can be used for. This requires grounding language-based action queries to objects that both afford the requested function and can be physically retrieved. Existing approaches often rely on black-box models or fixed affordance labels, limiting transparency, controllability, and reliability for human-facing applications. We introduce CRAFT-E, a modular neuro-symbolic framework that composes a structured verb-property-object knowledge graph with visual-language alignment and energy-based grasp reasoning. The system generates interpretable grounding paths that expose the factors influencing object selection and incorporates grasp feasibility as an integral part of affordance inference. We further construct a benchmark dataset with unified annotations for verb-object compatibility, segmentation, and grasp candidates, and deploy the full pipeline on a physical robot. CRAFT-E achieves competitive performance in static scenes, ImageNet-based functional retrieval, and real-world trials involving 20 verbs and 39 objects. The framework remains robust under perceptual noise and provides transparent, component-level diagnostics. By coupling symbolic reasoning with embodied perception, CRAFT-E offers an interpretable and customizable alternative to end-to-end models for affordance-grounded object selection, supporting trustworthy decision-making in assistive robotic systems.