# MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis

arXiv: https://arxiv.org/abs/2208.03963 · HTML: https://mlsanity.com/paper/2208.03963/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 8 Aug 2022.

## Builds on (most cited first, 22 of 22)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [ACRONYM: A Large-Scale Grasp Dataset Based on Simulation](https://mlsanity.com/paper/2011.09584/) (2020)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [EGAD! an Evolved Grasping Analysis Dataset for diversity and reproducibility in robotic manipulation](https://mlsanity.com/paper/2003.01314/) (2020)
- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images](https://mlsanity.com/paper/2103.02184/) (2021)
- [SuctionNet-1Billion: A Large-Scale Benchmark for Suction Grasping](https://mlsanity.com/paper/2103.12311/) (2021)
- [ROI-based Robotic Grasp Detection for Object Overlapping Scenes](https://mlsanity.com/paper/1808.10313/) (2018)
- [A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set](https://mlsanity.com/paper/1912.05604/) (2019)
- [OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation](https://mlsanity.com/paper/2104.11446/) (2021)
- [Dex-Net 3.0: Computing Robust Robot Vacuum Suction Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning](https://mlsanity.com/paper/1709.06670/) (2017)
- [REGRAD: A Large-Scale Relational Grasp Dataset for Safe and Object-Specific Robotic Grasping in Clutter](https://mlsanity.com/paper/2104.14118/) (2021)
- [Transferring Experience from Simulation to the Real World for Precise Pick-And-Place Tasks in Highly Cluttered Scenes](https://mlsanity.com/paper/2101.04781/) (2021)
- [SynPick: A Dataset for Dynamic Bin Picking Scene Understanding](https://mlsanity.com/paper/2107.04852/) (2021)

## Built on by (most cited first, 13 of 13)

- [Grasp-Anything: Large-scale Grasp Dataset from Foundation Models](https://mlsanity.com/paper/2309.09818/) (2023)
- [Language-driven Grasp Detection](https://mlsanity.com/paper/2406.09489/) (2024)
- [Fast GraspNeXt: A Fast Self-Attention Neural Network Architecture for Multi-task Learning in Computer Vision Tasks for Robotic Grasping on the Edge](https://mlsanity.com/paper/2304.11196/) (2023)
- [Uncertainty-driven Exploration Strategies for Online Grasp Learning](https://mlsanity.com/paper/2309.12038/) (2023)
- [Is Single-View Mesh Reconstruction Ready for Robotics?](https://mlsanity.com/paper/2505.17966/) (2025)
- [A Collision-Aware Cable Grasping Method in Cluttered Environment](https://mlsanity.com/paper/2402.14498/) (2024)
- [A Modern Take on Visual Relationship Reasoning for Grasp Planning](https://mlsanity.com/paper/2409.02035/) (2024)
- [QDGset: A Large Scale Grasping Dataset Generated with Quality-Diversity](https://mlsanity.com/paper/2410.02319/) (2024)
- [Corner-Grasp: Multi-Action Grasp Detection and Active Gripper Adaptation for Grasping in Cluttered Environments](https://mlsanity.com/paper/2504.01861/) (2025)
- [SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes](https://mlsanity.com/paper/2307.07333/) (2023)
- [RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation](https://mlsanity.com/paper/2603.14880/) (2026)
- [MMRNet: Improving Reliability for Multimodal Object Detection and Segmentation for Bin Picking via Multimodal Redundancy](https://mlsanity.com/paper/2210.10842/) (2022)
- [DA-Fusion: Deformable Attention-Based RGB-D Fusion Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2607.17754/) (2026)

## Abstract (arXiv)

Autonomous bin picking poses significant challenges to vision-driven robotic systems given the complexity of the problem, ranging from various sensor modalities, to highly entangled object layouts, to diverse item properties and gripper types. Existing methods often address the problem from one perspective. Diverse items and complex bin scenes require diverse picking strategies together with advanced reasoning. As such, to build robust and effective machine-learning algorithms for solving this complex task requires significant amounts of comprehensive and high quality data. Collecting such data in real world would be too expensive and time prohibitive and therefore intractable from a scalability perspective. To tackle this big, diverse data problem, we take inspiration from the recent rise in the concept of metaverses, and introduce MetaGraspNet, a large-scale photo-realistic bin picking dataset constructed via physics-based metaverse synthesis. The proposed dataset contains 217k RGBD images across 82 different article types, with full annotations for object detection, amodal perception, keypoint detection, manipulation order and ambidextrous grasp labels for a parallel-jaw and vacuum gripper. We also provide a real dataset consisting of over 2.3k fully annotated high-quality RGBD images, divided into 5 levels of difficulties and an unseen object set to evaluate different object and layout properties. Finally, we conduct extensive experiments showing that our proposed vacuum seal model and synthetic dataset achieves state-of-the-art performance and generalizes to real world use-cases.