Language-driven Grasp Detection with Mask-guided Attention
Not on the map. Why?
Abstract (arXiv)
Grasp detection is an essential task in robotics with various industrial applications. However, traditional methods often struggle with occlusions and do not utilize language for grasping. Incorporating natural language into grasp detection remains a challenging task and largely unexplored. To address this gap, we propose a new method for language-driven grasp detection with mask-guided attention by utilizing the transformer attention mechanism with semantic segmentation features. Our approach integrates visual data, segmentation mask features, and natural language instructions, significantly improving grasp detection accuracy. Our work introduces a new framework for language-driven grasp detection, paving the way for language-driven robotic applications. Intensive experiments show that our method outperforms other recent baselines by a clear margin, with a 10.0% success score improvement. We further validate our method in real-world robotic experiments, confirming the effectiveness of our approach.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 54 of its 69 references that are in the map. First posted on arXiv 29 Jul 2024.
Builds on
54 of 69 references.
References · 54
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2021ViViT: A Video Vision Transformercited by 1,610
- 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
- 2020Point Transformercited by 1,462
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
38 more
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
- 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
- 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
- 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
- 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
- 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
- 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
- 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
- 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
- 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
- 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
- 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
- 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
- 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
- 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
- 2013Deep Learning for Detecting Robotic Graspscited by 114
- 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
- 2018Jacquard: A Large Scale Dataset for Robotic Grasp Detectioncited by 86
- 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
- 2020Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentationcited by 64
- 2019Antipodal Robotic Grasping using Generative Residual Convolutional Neural Networkcited by 59
- 2021End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGBcited by 44
- 2021Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modelingcited by 43
- 2023Open-Vocabulary Affordance Detection in 3D Point Cloudscited by 41
- 2023VL-Grasp: a 6-Dof Interactive Grasp Policy for Language-Oriented Objects in Cluttered Indoor Scenescited by 40
- 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
- 2023Grasp-Anything: Large-scale Grasp Dataset from Foundation Modelscited by 33
- 2018ROI-based Robotic Grasp Detection for Object Overlapping Scenescited by 31
- 2024Language-driven Grasp Detectioncited by 26
- 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
- 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
- 2022Interactive Robotic Grasping with Attribute-Guided Disambiguationcited by 19
- 2022Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructionscited by 19
- 2023Open-Vocabulary Affordance Detection using Knowledge Distillation and Text-Point Correlationcited by 18
- 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
- 2022When Transformer Meets Robotic Grasping: Exploits Context for Efficient Grasp Detectioncited by 12
- 2023EC^2: Emergent Communication for Embodied Controlcited by 10
- 2021Research Challenges and Progress in Robotic Grasping and Manipulation Competitionscited by 9
50 most cited of 54 All 54 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.