# Language-driven Grasp Detection

arXiv: https://arxiv.org/abs/2406.09489 · HTML: https://mlsanity.com/paper/2406.09489/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 89 of its 110 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 89)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)

## Built on by (most cited first, 26 of 26)

- [Diffusion Models for Robotic Manipulation: A Survey](https://mlsanity.com/paper/2504.08438/) (2025)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability](https://mlsanity.com/paper/2503.08481/) (2025)
- [Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance](https://mlsanity.com/paper/2407.13842/) (2024)
- [Towards Open-World Grasping with Large Vision-Language Models](https://mlsanity.com/paper/2406.18722/) (2024)
- [RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping](https://mlsanity.com/paper/2507.23734/) (2025)
- [Lightweight Language-driven Grasp Detection using Conditional Consistency Model](https://mlsanity.com/paper/2407.17967/) (2024)
- [HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models](https://mlsanity.com/paper/2409.10419/) (2024)
- [Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting](https://mlsanity.com/paper/2512.20014/) (2025)
- [Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications](https://mlsanity.com/paper/2409.17727/) (2024)
- [GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learning](https://mlsanity.com/paper/2409.14403/) (2024)
- [Language-driven Grasp Detection with Mask-guided Attention](https://mlsanity.com/paper/2407.19877/) (2024)
- [VCoT-Grasp: Grasp Foundation Models with Visual Chain-of-Thought Reasoning for Language-driven Grasp Generation](https://mlsanity.com/paper/2510.05827/) (2025)
- [MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representation](https://mlsanity.com/paper/2602.07082/) (2026)
- [Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulation](https://mlsanity.com/paper/2512.21065/) (2025)
- [RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation](https://mlsanity.com/paper/2603.14880/) (2026)
- [ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Models](https://mlsanity.com/paper/2505.08417/) (2025)
- [Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoning](https://mlsanity.com/paper/2509.08126/) (2025)
- [XGrasp: Gripper-Aware Grasp Detection with Multi-Gripper Data Generation](https://mlsanity.com/paper/2510.11036/) (2025)
- [VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models](https://mlsanity.com/paper/2511.05791/) (2025)
- [AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifiers](https://mlsanity.com/paper/2603.27970/) (2026)
- [Object Pose and Shape Estimation for Grasping: Does it Work?](https://mlsanity.com/paper/2605.26944/) (2026)
- [DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation](https://mlsanity.com/paper/2607.04779/) (2026)
- [Beyond Visual Grasping: Benchmarking Complex Grasping from Detection to Execution](https://mlsanity.com/paper/2607.14341/) (2026)
- [Speech2Grasp: Data-Efficient Transfer of Text-Conditioned Grasp Detection to Speech in Humanoid Robots](https://mlsanity.com/paper/2607.26567/) (2026)
- [Gripper-aware Vision Language Action Models](https://mlsanity.com/paper/2608.24603/) (2026)

## Abstract (arXiv)

Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work. Project website: https://airvlab.github.io/grasp-anything/