# Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects

arXiv: https://arxiv.org/abs/2208.03792 · HTML: https://mlsanity.com/paper/2208.03792/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 7 Aug 2022.

## Builds on (most cited first, 29 of 29)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data](https://mlsanity.com/paper/1909.00889/) (2019)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations](https://mlsanity.com/paper/2107.14483/) (2021)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds](https://mlsanity.com/paper/2104.03437/) (2021)
- [Sparse and noisy LiDAR completion with RGB guidance and uncertainty](https://mlsanity.com/paper/1902.05356/) (2019)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [DeceptionNet: Network-Driven Domain Randomization](https://mlsanity.com/paper/1904.02750/) (2019)
- [Adaptive Surface Normal Constraint for Depth Estimation](https://mlsanity.com/paper/2103.15483/) (2021)
- [Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects](https://mlsanity.com/paper/2110.00087/) (2021)
- [Domain Randomization for Scene-Specific Car Detection and Pose Estimation](https://mlsanity.com/paper/1811.05939/) (2018)
- [Bayesian Deep Basis Fitting for Depth Completion with Uncertainty](https://mlsanity.com/paper/2103.15254/) (2021)
- [DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition](https://mlsanity.com/paper/1702.08558/) (2017)
- [Physics-based Differentiable Depth Sensor Simulation](https://mlsanity.com/paper/2103.16563/) (2021)

## Built on by (most cited first, 30 of 50)

- [GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF](https://mlsanity.com/paper/2210.06575/) (2022)
- [RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning](https://mlsanity.com/paper/2504.18904/) (2025)
- [Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking](https://mlsanity.com/paper/2406.04316/) (2024)
- [GenPose: Generative Category-level Object Pose Estimation via Diffusion Models](https://mlsanity.com/paper/2306.10531/) (2023)
- [DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes](https://mlsanity.com/paper/2410.23004/) (2024)
- [Masked Depth Modeling for Spatial Perception](https://mlsanity.com/paper/2601.17895/) (2026)
- [Challenges for Monocular 6D Object Pose Estimation in Robotics](https://mlsanity.com/paper/2307.12172/) (2023)
- [StereoPose: Category-Level 6D Transparent Object Pose Estimation from Stereo Images via Back-View NOCS](https://mlsanity.com/paper/2211.01644/) (2022)
- [TRansPose: Large-Scale Multispectral Dataset for Transparent Object](https://mlsanity.com/paper/2307.05016/) (2023)
- [Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots](https://mlsanity.com/paper/2509.02530/) (2025)
- [Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation](https://mlsanity.com/paper/2512.23705/) (2025)
- [ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera](https://mlsanity.com/paper/2405.05648/) (2024)
- [ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation](https://mlsanity.com/paper/2409.08926/) (2024)
- [Diffusion-Based Depth Inpainting for Transparent and Reflective Objects](https://mlsanity.com/paper/2410.08567/) (2024)
- [Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion](https://mlsanity.com/paper/2502.14616/) (2025)
- [RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface Normal Estimation and Manipulation](https://mlsanity.com/paper/2311.12398/) (2023)
- [Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds](https://mlsanity.com/paper/2211.16693/) (2022)
- [SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2403.19607/) (2024)
- [RaSim: A Range-aware High-fidelity RGB-D Data Simulation Pipeline for Real-world Applications](https://mlsanity.com/paper/2404.03962/) (2024)
- [GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulation](https://mlsanity.com/paper/2411.18276/) (2024)
- [Sim-to-Real Grasp Detection with Global-to-Local RGB-D Adaptation](https://mlsanity.com/paper/2403.11511/) (2024)
- [DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time](https://mlsanity.com/paper/2412.08053/) (2024)
- [Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models](https://mlsanity.com/paper/2605.29074/) (2026)
- [NeuGrasp: Generalizable Neural Surface Reconstruction with Background Priors for Material-Agnostic Object Grasp Detection](https://mlsanity.com/paper/2503.03511/) (2025)
- [TransTouch: Learning Transparent Objects Depth Sensing Through Sparse Touches](https://mlsanity.com/paper/2309.09427/) (2023)
- [STOPNet: Multiview-based 6-DoF Suction Detection for Transparent Objects on Production Lines](https://mlsanity.com/paper/2310.05717/) (2023)
- [TransNormal: Dense Visual Semantics for Diffusion-based Transparent Object Normal Estimation](https://mlsanity.com/paper/2602.00839/) (2026)
- [TransNet: Transparent Object Manipulation Through Category-Level Pose Estimation](https://mlsanity.com/paper/2307.12400/) (2023)
- [Depth Restoration of Hand-Held Transparent Objects for Human-to-Robot Handover](https://mlsanity.com/paper/2408.14997/) (2024)
- [QuadWBG: Generalizable Quadrupedal Whole-Body Grasping](https://mlsanity.com/paper/2411.06782/) (2024)

## Abstract (arXiv)

Commercial depth sensors usually generate noisy and missing depths, especially on specular and transparent objects, which poses critical issues to downstream depth or point cloud-based tasks. To mitigate this problem, we propose a powerful RGBD fusion network, SwinDRNet, for depth restoration. We further propose Domain Randomization-Enhanced Depth Simulation (DREDS) approach to simulate an active stereo depth system using physically based rendering and generate a large-scale synthetic dataset that contains 130K photorealistic RGB images along with their simulated depths carrying realistic sensor noises. To evaluate depth restoration methods, we also curate a real-world dataset, namely STD, that captures 30 cluttered scenes composed of 50 objects with different materials from specular, transparent, to diffuse. Experiments demonstrate that the proposed DREDS dataset bridges the sim-to-real domain gap such that, trained on DREDS, our SwinDRNet can seamlessly generalize to other real depth datasets, e.g. ClearGrasp, and outperform the competing methods on depth restoration with a real-time speed. We further show that our depth restoration effectively boosts the performance of downstream tasks, including category-level pose estimation and grasping tasks. Our data and code are available at https://github.com/PKU-EPIC/DREDS