mlsanityData to 7 Oct 2026

Language-driven Grasp Detection

arXiv 2406.09489 ↗First posted 13 Jun 2024cs.CVcited by 26

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work. Project website: https://airvlab.github.io/grasp-anything/

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 89 of its 110 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

89 of 110 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023EDMP: Ensemble-of-costs-guided Diffusion for Motion Planning20 Sep
  2. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models3 Aug
  3. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion8 Sep

Other references · 86

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Segment Anythingcited by 8,158
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2022Video Diffusion Modelscited by 2,280
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2021Vision Transformers for Dense Predictioncited by 1,789
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  12. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  13. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  14. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  15. 2022Human Motion Diffusion Modelcited by 1,062
  16. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  17. 2022Diffusion Models in Vision: A Surveycited by 1,008
  18. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  19. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  21. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  22. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  23. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  24. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  25. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  26. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  27. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  28. 2022EDGE: Editable Dance Generation From Musiccited by 284
  29. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  30. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  31. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  32. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  33. 2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
  34. 2020A Review of Generalized Zero-Shot Learning Methodscited by 159
  35. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  36. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Viewscited by 148
  37. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  38. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 8.

26 papers.

Sort:

By area:

  1. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  2. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  3. 2025PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachabilitycited by 17
  4. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  5. 2024Towards Open-World Grasping with Large Vision-Language Modelscited by 13
  6. 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11
  7. 2024Lightweight Language-driven Grasp Detection using Conditional Consistency Modelcited by 9
  8. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  9. 2024Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applicationscited by 5
  10. 2025Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Promptingcited by 5
  11. 2024GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learningcited by 3
  12. 2024Language-driven Grasp Detection with Mask-guided Attentioncited by 2
14 more
  1. 2025VCoT-Grasp: Grasp Foundation Models with Visual Chain-of-Thought Reasoning for Language-driven Grasp Generationcited by 2
  2. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  3. 2025Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulationcited by 1
  4. 2026RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulationcited by 1
  5. 2025ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Modelscited by 0
  6. 2025Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoningcited by 0
  7. 2025XGrasp: Gripper-Aware Grasp Detection with Multi-Gripper Data Generationcited by 0
  8. 2025VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Modelscited by 0
  9. 2026AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifierscited by 0
  10. 2026Object Pose and Shape Estimation for Grasping: Does it Work?cited by 0
  11. 2026DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentationcited by 0
  12. 2026Beyond Visual Grasping: Benchmarking Complex Grasping from Detection to Executioncited by 0
  13. 2026Speech2Grasp: Data-Efficient Transfer of Text-Conditioned Grasp Detection to Speech in Humanoid Robotscited by 0
  14. 2026Gripper-aware Vision Language Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
  2. 2025Training-free Task-oriented Grasp Generation
  3. 2024Reasoning Grasping via Multimodal Large Language Model
  4. 2025Free-form language-based robotic reasoning and grasping
  5. 2025Using Visual Language Models to Control Bionic Hands: Assessment of Object Perception and Grasp Inference
  6. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scale
  7. 2024Grasp Diffusion Network: Learning Grasp Generators from Partial Point Clouds with Diffusion Models in SO(3)xR3
  8. 2024Multi-GraspLLM: A Multimodal LLM for Multi-Hand Semantic Guided Grasp Generation