# PEGG-Net: Pixel-Wise Efficient Grasp Generation in Complex Scenes

arXiv: https://arxiv.org/abs/2203.16301 · HTML: https://mlsanity.com/paper/2203.16301/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [Real-Time Grasp Detection Using Convolutional Neural Networks](https://mlsanity.com/paper/1412.3128/) (2014)
- [Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network](https://mlsanity.com/paper/1909.04810/) (2019)
- [Robotic Grasp Detection using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1611.08036/) (2016)
- [Learning a visuomotor controller for real world robotic grasping using simulated depth images](https://mlsanity.com/paper/1706.04652/) (2017)
- [Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty](https://mlsanity.com/paper/1608.02239/) (2016)
- [DC-SPP-YOLO: Dense Connection and Spatial Pyramid Pooling Based YOLO for Object Detection](https://mlsanity.com/paper/1903.08589/) (2019)
- [GKNet: grasp keypoint network for grasp candidates detection](https://mlsanity.com/paper/2106.08497/) (2021)
- [Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks](https://mlsanity.com/paper/1906.08989/) (2019)
- [Efficient Fully Convolution Neural Network for Generating Pixel Wise Robotic Grasps With High Resolution Images](https://mlsanity.com/paper/1902.08950/) (2019)

## Abstract (arXiv)

Vision-based grasp estimation is an essential part of robotic manipulation tasks in the real world. Existing planar grasp estimation algorithms have been demonstrated to work well in relatively simple scenes. But when it comes to complex scenes, such as cluttered scenes with messy backgrounds and moving objects, the algorithms from previous works are prone to generate inaccurate and unstable grasping contact points. In this work, we first study the existing planar grasp estimation algorithms and analyze the related challenges in complex scenes. Secondly, we design a Pixel-wise Efficient Grasp Generation Network (PEGG-Net) to tackle the problem of grasping in complex scenes. PEGG-Net can achieve improved state-of-the-art performance on the Cornell dataset (98.9%) and second-best performance on the Jacquard dataset (93.8%), outperforming other existing algorithms without the introduction of complex structures. Thirdly, PEGG-Net could operate in a closed-loop manner for added robustness in dynamic environments using position-based visual servoing (PBVS). Finally, we conduct real-world experiments on static, dynamic, and cluttered objects in different complex scenes. The results show that our proposed network achieves a high success rate in grasping irregular objects, household objects, and workshop tools. To benefit the community, our trained model and supplementary materials are available at https://github.com/HZWang96/PEGG-Net.