mlsanityData to 7 Oct 2026

A Joint Network for Grasp Detection Conditioned on Natural Language Commands

arXiv 2104.00492 ↗First posted 1 Apr 2021cs.CV cs.ROcited by 23

Not on the map. Why?

Abstract (arXiv)

We consider the task of grasping a target object based on a natural language command query. Previous work primarily focused on localizing the object given the query, which requires a separate grasp detection module to grasp it. The cascaded application of two pipelines incurs errors in overlapping multi-object cases due to ambiguity in the individual outputs. This work proposes a model named Command Grasping Network(CGNet) to directly output command satisficing grasps from RGB image and textual command inputs. A dataset with ground truth (image, command, grasps) tuple is generated based on the VMRD dataset to train the proposed network. Experimental results on the generated test set show that CGNet outperforms a cascaded object-retrieval and grasp detection baseline by a large margin. Three physical experiments demonstrate the functionality and performance of CGNet.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

21 of 34 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Fast R-CNNcited by 2,930
  3. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  4. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  5. 2015Natural Language Object Retrievalcited by 124
  6. 2013Deep Learning for Detecting Robotic Graspscited by 114
  7. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  8. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  9. 2014Real-Time Grasp Detection Using Convolutional Neural Networkscited by 75
  10. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  11. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  12. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
9 more
  1. 2018Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervisioncited by 50
  2. 2016Robotic Grasp Detection using Deep Convolutional Neural Networkscited by 47
  3. 2018ROI-based Robotic Grasp Detection for Object Overlapping Scenescited by 31
  4. 2019Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Cluttercited by 30
  5. 2020Robot Object Retrieval with Contextual Natural Language Queriescited by 26
  6. 2017End-to-End Learning of Semantic Graspingcited by 20
  7. 2019CAGE: Context-Aware Grasping Enginecited by 9
  8. 2019Grounding Language Attributes to Objects using Bayesian Eigenobjectscited by 8
  9. 2018Object Captioning and Retrieval with Natural Languagecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 5; 2024: 6; 2025: 2; 2026: 1.

23 papers.

Sort:

By area:

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  4. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  5. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  6. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  7. 2024Reasoning Grasping via Multimodal Large Language Modelcited by 20
  8. 2022Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructionscited by 19
  9. 2023Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Groundingcited by 17
  10. 2022Keypoint-GraspNet: Keypoint-based 6-DoF Grasp Generation from the Monocular RGB-D inputcited by 10
  11. 2022Policy Adaptation from Foundation Model Feedbackcited by 10
  12. 2024Lightweight Language-driven Grasp Detection using Conditional Consistency Modelcited by 9
11 more
  1. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  2. 2023WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Modelcited by 6
  3. 2025Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model Systemcited by 5
  4. 2022I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketchescited by 4
  5. 2023KGNv2: Separating Scale and Pose Prediction for Keypoint-based 6-DoF Grasp Synthesis on RGB-D inputcited by 4
  6. 2024SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Imagescited by 3
  7. 2024Language-driven Grasp Detection with Mask-guided Attentioncited by 2
  8. 2024Target-Oriented Object Grasping via Multimodal Human Guidancecited by 2
  9. 2026VISTA: Enhancing Visual Conditioning via Track-Following Preference Optimization in Vision-Language-Action Modelscited by 2
  10. 2025Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulationcited by 1
  11. 2022Enhancing Interpretability and Interactivity in Robot Manipulation: A Neurosymbolic Approachcited by 0