# When Transformer Meets Robotic Grasping: Exploits Context for Efficient Grasp Detection

arXiv: https://arxiv.org/abs/2202.11911 · HTML: https://mlsanity.com/paper/2202.11911/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [Real-Time Grasp Detection Using Convolutional Neural Networks](https://mlsanity.com/paper/1412.3128/) (2014)
- [Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network](https://mlsanity.com/paper/1909.04810/) (2019)
- [Robotic Grasp Detection using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1611.08036/) (2016)
- [End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGB](https://mlsanity.com/paper/2107.05287/) (2021)
- [ROI-based Robotic Grasp Detection for Object Overlapping Scenes](https://mlsanity.com/paper/1808.10313/) (2018)
- [Fully Convolutional Grasp Detection Network with Oriented Anchor Box](https://mlsanity.com/paper/1803.02209/) (2018)
- [6-DoF Contrastive Grasp Proposal Network](https://mlsanity.com/paper/2103.15995/) (2021)
- [Noticing Motion Patterns: Temporal CNN with a Novel Convolution Operator for Human Trajectory Prediction](https://mlsanity.com/paper/2007.00862/) (2020)
- [GQ-STN: Optimizing One-Shot Grasp Detection based on Robustness Classifier](https://mlsanity.com/paper/1903.02489/) (2019)
- [Learn to Grasp with Less Supervision: A Data-Efficient Maximum Likelihood Grasp Sampling Loss](https://mlsanity.com/paper/2110.01379/) (2021)
- [Fusing Convolutional Neural Network and Geometric Constraint for Image-based Indoor Localization](https://mlsanity.com/paper/2201.01408/) (2022)

## Built on by (most cited first, 12 of 12)

- [Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding](https://mlsanity.com/paper/2301.11564/) (2023)
- [WorldAfford: Affordance Grounding based on Natural Language Instructions](https://mlsanity.com/paper/2405.12461/) (2024)
- [TODE-Trans: Transparent Object Depth Estimation with Transformer](https://mlsanity.com/paper/2209.08455/) (2022)
- [Large Scale Foundation Models for Intelligent Manufacturing Applications: A Survey](https://mlsanity.com/paper/2312.06718/) (2023)
- [NBMOD: Find It and Grasp It in Noisy Background](https://mlsanity.com/paper/2306.10265/) (2023)
- [A Robotic Visual Grasping Design: Rethinking Convolution Neural Network with High-Resolutions](https://mlsanity.com/paper/2209.07459/) (2022)
- [Language-driven Grasp Detection with Mask-guided Attention](https://mlsanity.com/paper/2407.19877/) (2024)
- [Relevance for Human Robot Collaboration](https://mlsanity.com/paper/2409.07753/) (2024)
- [HMT-Grasp: A Hybrid Mamba-Transformer Approach for Robot Grasping in Cluttered Environments](https://mlsanity.com/paper/2410.03522/) (2024)
- [Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulation](https://mlsanity.com/paper/2512.21065/) (2025)
- [GraspCaps: A Capsule Network Approach for Familiar 6DoF Object Grasping](https://mlsanity.com/paper/2210.03628/) (2022)
- [XGrasp: Gripper-Aware Grasp Detection with Multi-Gripper Data Generation](https://mlsanity.com/paper/2510.11036/) (2025)

## Abstract (arXiv)

In this paper, we present a transformer-based architecture, namely TF-Grasp, for robotic grasp detection. The developed TF-Grasp framework has two elaborate designs making it well suitable for visual grasping tasks. The first key design is that we adopt the local window attention to capture local contextual information and detailed features of graspable objects. Then, we apply the cross window attention to model the long-term dependencies between distant pixels. Object knowledge, environmental configuration, and relationships between different visual entities are aggregated for subsequent grasp detection. The second key design is that we build a hierarchical encoder-decoder architecture with skip-connections, delivering shallow features from encoder to decoder to enable a multi-scale feature fusion. Due to the powerful attention mechanism, the TF-Grasp can simultaneously obtain the local information (i.e., the contours of objects), and model long-term connections such as the relationships between distinct visual concepts in clutter. Extensive computational experiments demonstrate that the TF-Grasp achieves superior results versus state-of-art grasping convolutional models and attain a higher accuracy of 97.99% and 94.6% on Cornell and Jacquard grasping datasets, respectively. Real-world experiments using a 7DoF Franka Emika Panda robot also demonstrate its capability of grasping unseen objects in a variety of scenarios. The code and pre-trained models will be available at https://github.com/WangShaoSUN/grasp-transformer