# Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation

arXiv: https://arxiv.org/abs/2407.12449 · HTML: https://mlsanity.com/paper/2407.12449/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 14 of 14)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes](https://mlsanity.com/paper/1707.04796/) (2017)
- [NViSII: A Scriptable Tool for Photorealistic Image Generation](https://mlsanity.com/paper/2105.13962/) (2021)
- [DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition](https://mlsanity.com/paper/1702.08558/) (2017)
- [BlendTorch: A Real-Time, Adaptive Domain Randomization Library](https://mlsanity.com/paper/2010.11696/) (2020)
- [Keep it Unreal: Bridging the Realism Gap for 2.5D Recognition with Geometry Priors Only](https://mlsanity.com/paper/1804.09113/) (2018)
- [Cluttered Food Grasping with Adaptive Fingers and Synthetic-Data Trained Object Detection](https://mlsanity.com/paper/2203.05187/) (2022)
- [Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data](https://mlsanity.com/paper/1809.05825/) (2018)
- [Synthetic dataset generation for object-to-model deep learning in industrial applications](https://mlsanity.com/paper/1909.10976/) (2019)
- [StandardSim: A Synthetic Dataset For Retail Environments](https://mlsanity.com/paper/2202.02418/) (2022)
- [Sim-to-Real Transfer of Robotic Assembly with Visual Inputs Using CycleGAN and Force Control](https://mlsanity.com/paper/2208.14104/) (2022)

## Built on by (most cited first, 6 of 6)

- [MetaSym: A Symplectic Meta-learning Framework for Physical Intelligence](https://mlsanity.com/paper/2502.16667/) (2025)
- [FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulation](https://mlsanity.com/paper/2509.19102/) (2025)
- [Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion](https://mlsanity.com/paper/2507.23483/) (2025)
- [VIRTUS-FPP: Virtual Sensor Modeling for Fringe Projection Profilometry in NVIDIA Isaac Sim](https://mlsanity.com/paper/2509.22685/) (2025)
- [Ontology-Guided Diffusion for Zero-Shot Visual Sim2Real Transfer](https://mlsanity.com/paper/2603.18719/) (2026)
- [A Synthetic Data Pipeline for Supporting Manufacturing SMEs in Visual Assembly Control](https://mlsanity.com/paper/2509.13089/) (2025)

## Abstract (arXiv)

Despite the substantial progress in deep learning, its adoption in industrial robotics projects remains limited, primarily due to challenges in data acquisition and labeling. Previous sim2real approaches using domain randomization require extensive scene and model optimization. To address these issues, we introduce an innovative physically-based structured light simulation system, generating both RGB and physically realistic depth images, surpassing previous dataset generation tools. We create an RGBD dataset tailored for robotic industrial grasping scenarios and evaluate it across various tasks, including object detection, instance segmentation, and embedding sim2real visual perception in industrial robotic grasping. By reducing the sim2real gap and enhancing deep learning training, we facilitate the application of deep learning models in industrial settings. Project details are available at https://baikaixinpublic.github.io/structured light 3D synthesizer/.