# DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition

arXiv: https://arxiv.org/abs/1702.08558 · HTML: https://mlsanity.com/paper/1702.08558/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2017.

## Built on by (most cited first, 15 of 15)

- [Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects](https://mlsanity.com/paper/2208.03792/) (2022)
- [LRM-Zero: Training Large Reconstruction Models with Synthesized Data](https://mlsanity.com/paper/2406.09371/) (2024)
- [BodyPressure -- Inferring Body Pose and Contact Pressure from a Depth Image](https://mlsanity.com/paper/2105.09936/) (2021)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [Physics-based Differentiable Depth Sensor Simulation](https://mlsanity.com/paper/2103.16563/) (2021)
- [Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation](https://mlsanity.com/paper/2407.12449/) (2024)
- [Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap](https://mlsanity.com/paper/2203.03833/) (2022)
- [The Drunkard's Odometry: Estimating Camera Motion in Deforming Scenes](https://mlsanity.com/paper/2306.16917/) (2023)
- [Active-Passive SimStereo -- Benchmarking the Cross-Generalization Capabilities of Deep Learning-based Stereo Methods](https://mlsanity.com/paper/2209.08305/) (2022)
- [DCL: Differential Contrastive Learning for Geometry-Aware Depth Synthesis](https://mlsanity.com/paper/2107.13087/) (2021)
- [Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion](https://mlsanity.com/paper/2507.23483/) (2025)
- [Enhancing Object Detection Performance for Small Objects through Synthetic Data Generation and Proportional Class-Balancing Technique: A Comparative Study in Industrial Scenarios](https://mlsanity.com/paper/2401.12729/) (2024)
- [G3R: Generating Rich and Fine-grained mmWave Radar Data from 2D Videos for Generalized Gesture Recognition](https://mlsanity.com/paper/2404.14934/) (2024)
- [Accurate Ground-Truth Depth Image Generation via Overfit Training of Point Cloud Registration using Local Frame Sets](https://mlsanity.com/paper/2207.07016/) (2022)
- [Sim-to-Real Deep Reinforcement Learning with Manipulators for Pick-and-place](https://mlsanity.com/paper/2309.09247/) (2023)

## Abstract (arXiv)

Recent progress in computer vision has been dominated by deep neural networks trained over large amounts of labeled data. Collecting such datasets is however a tedious, often impossible task; hence a surge in approaches relying solely on synthetic data for their training. For depth images however, discrepancies with real scans still noticeably affect the end performance. We thus propose an end-to-end framework which simulates the whole mechanism of these devices, generating realistic depth data from 3D models by comprehensively modeling vital factors e.g. sensor noise, material reflectance, surface geometry. Not only does our solution cover a wider range of sensors and achieve more realistic results than previous methods, assessed through extended evaluation, but we go further by measuring the impact on the training of neural networks for various recognition tasks; demonstrating how our pipeline seamlessly integrates such architectures and consistently enhances their performance.