# Lan-grasp: Using Large Language Models for Semantic Object Grasping and Placement

arXiv: https://arxiv.org/abs/2310.05239 · HTML: https://mlsanity.com/paper/2310.05239/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2023.

## Built on by (most cited first, 16 of 16)

- [RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis](https://mlsanity.com/paper/2402.16117/) (2024)
- [One-Shot Open Affordance Learning with Foundation Models](https://mlsanity.com/paper/2311.17776/) (2023)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [Reasoning Grasping via Multimodal Large Language Model](https://mlsanity.com/paper/2402.06798/) (2024)
- [ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decomposition](https://mlsanity.com/paper/2403.18062/) (2024)
- [HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models](https://mlsanity.com/paper/2409.10419/) (2024)
- [IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Models](https://mlsanity.com/paper/2503.10110/) (2025)
- [The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Study](https://mlsanity.com/paper/2504.04540/) (2025)
- [Constrained 6-DoF Grasp Generation on Complex Shapes for Improved Dual-Arm Manipulation](https://mlsanity.com/paper/2404.04643/) (2024)
- [GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learning](https://mlsanity.com/paper/2409.14403/) (2024)
- [HGDiffuser: Efficient Task-Oriented Grasp Generation via Human-Guided Grasp Diffusion Models](https://mlsanity.com/paper/2503.00508/) (2025)
- [Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis](https://mlsanity.com/paper/2609.04096/) (2026)
- [ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Models](https://mlsanity.com/paper/2505.08417/) (2025)
- [Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning](https://mlsanity.com/paper/2604.08726/) (2026)
- [SAGP: Semantic Affordance-Guided Grasp Planning via Coarse-Zone VLM Reasoning](https://mlsanity.com/paper/2607.29374/) (2026)
- [Zero-shot 2D Grounding with Novel Affordance Types](https://mlsanity.com/paper/2608.08929/) (2026)

## Abstract (arXiv)

In this paper, we propose Lan-grasp, a novel approach towards more appropriate semantic grasping and placing. We leverage foundation models to equip the robot with a semantic understanding of object geometry, enabling it to identify the right place to grasp, which parts to avoid, and the natural pose for placement. This is an important contribution to grasping and utilizing objects in a more meaningful and safe manner. We leverage a combination of a Large Language Model, a Vision-Language Model, and a traditional grasp planner to generate grasps that demonstrate a deeper semantic understanding of the objects. Building on foundation models provides us with a zero-shot grasp method that can handle a wide range of objects without requiring further training or fine-tuning. We also propose a method for safely putting down a grasped object. The core idea is to rotate the object upright utilizing a pretrained generative model and the reasoning capabilities of a VLM. We evaluate our method in real-world experiments on a custom object dataset and present the results of a survey that asks participants to choose an object part appropriate for grasping. The results show that the grasps generated by our method are consistently ranked higher by the participants than those generated by a conventional grasping planner and a recent semantic grasping approach. In addition, we propose a Visual Chain-of-Thought feedback loop to assess grasp feasibility in complex scenarios. This mechanism enables dynamic reasoning and generates alternative grasp strategies when needed, ensuring safer and more effective grasping outcomes.