# Keep it Unreal: Bridging the Realism Gap for 2.5D Recognition with Geometry Priors Only

arXiv: https://arxiv.org/abs/1804.09113 · HTML: https://mlsanity.com/paper/1804.09113/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2018.

## Built on by (most cited first, 9 of 9)

- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [DPODv2: Dense Correspondence-Based 6 DoF Pose Estimation](https://mlsanity.com/paper/2207.02805/) (2022)
- [BodyPressure -- Inferring Body Pose and Contact Pressure from a Depth Image](https://mlsanity.com/paper/2105.09936/) (2021)
- [Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces](https://mlsanity.com/paper/2105.11283/) (2021)
- [Physics-based Differentiable Depth Sensor Simulation](https://mlsanity.com/paper/2103.16563/) (2021)
- [Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation](https://mlsanity.com/paper/2407.12449/) (2024)
- [Photo-realistic Neural Domain Randomization](https://mlsanity.com/paper/2210.12682/) (2022)
- [6IMPOSE: Bridging the Reality Gap in 6D Pose Estimation for Robotic Grasping](https://mlsanity.com/paper/2208.14288/) (2022)
- [High-Quality Unknown Object Instance Segmentation via Quadruple Boundary Error Refinement](https://mlsanity.com/paper/2306.16132/) (2023)

## Abstract (arXiv)

With the increasing availability of large databases of 3D CAD models, depth-based recognition methods can be trained on an uncountable number of synthetically rendered images. However, discrepancies with the real data acquired from various depth sensors still noticeably impede progress. Previous works adopted unsupervised approaches to generate more realistic depth data, but they all require real scans for training, even if unlabeled. This still represents a strong requirement, especially when considering real-life/industrial settings where real training images are hard or impossible to acquire, but texture-less 3D models are available. We thus propose a novel approach leveraging only CAD models to bridge the realism gap. Purely trained on synthetic data, playing against an extensive augmentation pipeline in an unsupervised manner, our generative adversarial network learns to effectively segment depth images and recover the clean synthetic-looking depth information even from partial occlusions. As our solution is not only fully decoupled from the real domains but also from the task-specific analytics, the pre-processed scans can be handed to any kind and number of recognition methods also trained on synthetic data. Through various experiments, we demonstrate how this simplifies their training and consistently enhances their performance, with results on par with the same methods trained on real data, and better than usual approaches doing the reverse mapping.