# Photorealistic Image Synthesis for Object Instance Detection

arXiv: https://arxiv.org/abs/1902.03334 · HTML: https://mlsanity.com/paper/1902.03334/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2019.

## Built on by (most cited first, 30 of 66)

- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [BOP Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objects](https://mlsanity.com/paper/2302.13075/) (2023)
- [Multi-Modal 3D Object Detection in Autonomous Driving: a Survey](https://mlsanity.com/paper/2106.12735/) (2021)
- [BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects](https://mlsanity.com/paper/2403.09799/) (2024)
- [Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Data](https://mlsanity.com/paper/2311.18729/) (2023)
- [X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusion](https://mlsanity.com/paper/2212.03863/) (2022)
- [Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking](https://mlsanity.com/paper/2406.09598/) (2024)
- [Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation](https://mlsanity.com/paper/2203.10339/) (2022)
- [Perspective Flow Aggregation for Data-Limited 6D Object Pose Estimation](https://mlsanity.com/paper/2203.09836/) (2022)
- [Neural-Sim: Learning to Generate Training Data with NeRF](https://mlsanity.com/paper/2207.11368/) (2022)
- [ZePHyR: Zero-shot Pose Hypothesis Rating](https://mlsanity.com/paper/2104.13526/) (2021)
- [Challenges for Monocular 6D Object Pose Estimation in Robotics](https://mlsanity.com/paper/2307.12172/) (2023)
- [DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstration](https://mlsanity.com/paper/2112.02849/) (2021)
- [DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detection](https://mlsanity.com/paper/2206.09592/) (2022)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds](https://mlsanity.com/paper/2103.01977/) (2021)
- [A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?](https://mlsanity.com/paper/2108.11018/) (2021)
- [DSC-PoseNet: Learning 6DoF Object Pose Estimation via Dual-scale Consistency](https://mlsanity.com/paper/2104.03658/) (2021)
- [PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation](https://mlsanity.com/paper/2401.02281/) (2024)
- [COPE: End-to-end trainable Constant Runtime Object Pose Estimation](https://mlsanity.com/paper/2208.08807/) (2022)
- [Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation](https://mlsanity.com/paper/2309.05956/) (2023)
- [Analysis of Training Object Detection Models with Synthetic Data](https://mlsanity.com/paper/2211.16066/) (2022)
- [Bayesian Active Learning for Sim-to-Real Robotic Perception](https://mlsanity.com/paper/2109.11547/) (2021)
- [OSSID: Online Self-Supervised Instance Detection by (and for) Pose Estimation](https://mlsanity.com/paper/2201.07309/) (2022)
- [Dataset of Industrial Metal Objects](https://mlsanity.com/paper/2208.04052/) (2022)
- [TransPose: 6D Object Pose Estimation with Geometry-Aware Transformer](https://mlsanity.com/paper/2310.16279/) (2023)
- [Physics-based Differentiable Depth Sensor Simulation](https://mlsanity.com/paper/2103.16563/) (2021)
- [Ambiguity-Aware Multi-Object Pose Optimization for Visually-Assisted Robot Manipulation](https://mlsanity.com/paper/2211.00960/) (2022)

## Abstract (arXiv)

We present an approach to synthesize highly photorealistic images of 3D object models, which we use to train a convolutional neural network for detecting the objects in real images. The proposed approach has three key ingredients: (1) 3D object models are rendered in 3D models of complete scenes with realistic materials and lighting, (2) plausible geometric configuration of objects and cameras in a scene is generated using physics simulations, and (3) high photorealism of the synthesized images achieved by physically based rendering. When trained on images synthesized by the proposed approach, the Faster R-CNN object detector achieves a 24% absolute improvement of mAP@.75IoU on Rutgers APC and 11% on LineMod-Occluded datasets, compared to a baseline where the training images are synthesized by rendering object models on top of random photographs. This work is a step towards being able to effectively train object detectors without capturing or annotating any real images. A dataset of 600K synthetic images with ground truth annotations for various computer vision tasks will be released on the project website: thodan.github.io/objectsynth.