# ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decomposition

arXiv: https://arxiv.org/abs/2403.18062 · HTML: https://mlsanity.com/paper/2403.18062/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Language-Conditioned Imitation Learning for Robot Manipulation Tasks](https://mlsanity.com/paper/2010.12083/) (2020)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping](https://mlsanity.com/paper/2309.07970/) (2023)
- [Learning Unsupervised Hierarchical Part Decomposition of 3D Objects from a Single RGB Image](https://mlsanity.com/paper/2004.01176/) (2020)
- [Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Search](https://mlsanity.com/paper/2205.02961/) (2022)
- [Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision](https://mlsanity.com/paper/1806.09266/) (2018)
- [GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping](https://mlsanity.com/paper/2307.13204/) (2023)
- [Dexterous Functional Grasping](https://mlsanity.com/paper/2312.02975/) (2023)
- [OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics](https://mlsanity.com/paper/2401.12202/) (2024)
- [Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping](https://mlsanity.com/paper/2011.06431/) (2020)
- [SKILL: Structured Knowledge Infusion for Large Language Models](https://mlsanity.com/paper/2205.08184/) (2022)
- [End-to-End Learning of Semantic Grasping](https://mlsanity.com/paper/1707.01932/) (2017)
- [Lan-grasp: Using Large Language Models for Semantic Object Grasping and Placement](https://mlsanity.com/paper/2310.05239/) (2023)
- [End-to-End Learning to Grasp via Sampling from Object Point Clouds](https://mlsanity.com/paper/2203.05585/) (2022)
- [Robotic Grasping from Classical to Modern: A Survey](https://mlsanity.com/paper/2202.03631/) (2022)
- [Task-Oriented Grasp Prediction with Visual-Language Inputs](https://mlsanity.com/paper/2302.14355/) (2023)
- [Physical Primitive Decomposition](https://mlsanity.com/paper/1809.05070/) (2018)
- [Sample-Efficient Learning of Novel Visual Concepts](https://mlsanity.com/paper/2306.09482/) (2023)
- [Using Synthetic Data and Deep Networks to Recognize Primitive Shapes for Object Grasping](https://mlsanity.com/paper/1909.08508/) (2019)
- [Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos](https://mlsanity.com/paper/2309.05943/) (2023)

## Built on by (most cited first, 14 of 14)

- [AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter](https://mlsanity.com/paper/2503.00778/) (2025)
- [GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Grasping](https://mlsanity.com/paper/2411.12286/) (2024)
- [VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing](https://mlsanity.com/paper/2605.30117/) (2026)
- [Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions](https://mlsanity.com/paper/2505.05517/) (2025)
- [LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation](https://mlsanity.com/paper/2411.00773/) (2024)
- [HGDiffuser: Efficient Task-Oriented Grasp Generation via Human-Guided Grasp Diffusion Models](https://mlsanity.com/paper/2503.00508/) (2025)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [GraspCoT: Integrating Physical Property Reasoning for 6-DoF Grasping under Flexible Language Instructions](https://mlsanity.com/paper/2503.16013/) (2025)
- [Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping](https://mlsanity.com/paper/2508.03099/) (2025)
- [EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models](https://mlsanity.com/paper/2605.17070/) (2026)
- [ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Models](https://mlsanity.com/paper/2505.08417/) (2025)
- [Learning to Grasp Anything by Playing with Random Toys](https://mlsanity.com/paper/2510.12866/) (2025)
- [VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models](https://mlsanity.com/paper/2511.05791/) (2025)
- [SECOND-Grasp: Semantic Contact-guided Dexterous Grasping](https://mlsanity.com/paper/2605.13117/) (2026)

## Abstract (arXiv)

Task-oriented grasping of unfamiliar objects is a necessary skill for robots in dynamic in-home environments. Inspired by the human capability to grasp such objects through intuition about their shape and structure, we present a novel zero-shot task-oriented grasping method leveraging a geometric decomposition of the target object into simple, convex shapes that we represent in a graph structure, including geometric attributes and spatial relationships. Our approach employs minimal essential information - the object's name and the intended task - to facilitate zero-shot task-oriented grasping. We utilize the commonsense reasoning capabilities of large language models to dynamically assign semantic meaning to each decomposed part and subsequently reason over the utility of each part for the intended task. Through extensive experiments on a real-world robotics platform, we demonstrate that our grasping approach's decomposition and reasoning pipeline is capable of selecting the correct part in 92% of the cases and successfully grasping the object in 82% of the tasks we evaluate. Additional videos, experiments, code, and data are available on our project website: https://shapegrasp.github.io/.