# Synthetic Data Generation for Bridging Sim2Real Gap in a Production Environment

arXiv: https://arxiv.org/abs/2311.11039 · HTML: https://mlsanity.com/paper/2311.11039/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 18 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation](https://mlsanity.com/paper/1804.06534/) (2018)
- [Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks](https://mlsanity.com/paper/1512.04143/) (2015)
- [BlenderProc](https://mlsanity.com/paper/1911.01911/) (2019)
- [Photorealistic Image Synthesis for Object Instance Detection](https://mlsanity.com/paper/1902.03334/) (2019)
- [How much real data do we actually need: Analyzing object detection performance using synthetic and real data](https://mlsanity.com/paper/1907.07061/) (2019)
- [NViSII: A Scriptable Tool for Photorealistic Image Generation](https://mlsanity.com/paper/2105.13962/) (2021)
- [CAD2Render: A Modular Toolkit for GPU-accelerated Photorealistic Synthetic Data Generation for the Manufacturing Industry](https://mlsanity.com/paper/2211.14054/) (2022)
- [Applying Domain Randomization to Synthetic Data for Object Category Detection](https://mlsanity.com/paper/1807.09834/) (2018)
- [Style-transfer GANs for bridging the domain gap in synthetic pose estimator training](https://mlsanity.com/paper/2004.13681/) (2020)
- [Synthetic Data for Object Classification in Industrial Applications](https://mlsanity.com/paper/2212.04790/) (2022)

## Built on by (most cited first, 4 of 4)

- [Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis](https://mlsanity.com/paper/2508.05580/) (2025)
- [MultiFloodSynth: Multi-Annotated Flood Synthetic Dataset Generation](https://mlsanity.com/paper/2502.03966/) (2025)
- [The Synthetic Mirror -- Synthetic Data at the Age of Agentic AI](https://mlsanity.com/paper/2506.13818/) (2025)
- [A Synthetic-Driven Vision System for Assembly Step Recognition](https://mlsanity.com/paper/2607.00129/) (2026)

## Abstract (arXiv)

Synthetic data is being used lately for training deep neural networks in computer vision applications such as object detection, object segmentation and 6D object pose estimation. Domain randomization hereby plays an important role in reducing the simulation to reality gap. However, this generalization might not be effective in specialized domains like a production environment involving complex assemblies. Either the individual parts, trained with synthetic images, are integrated in much larger assemblies making them indistinguishable from their counterparts and result in false positives or are partially occluded just enough to give rise to false negatives. Domain knowledge is vital in these cases and if conceived effectively while generating synthetic data, can show a considerable improvement in bridging the simulation to reality gap. This paper focuses on synthetic data generation procedures for parts and assemblies used in a production environment. The basic procedures for synthetic data generation and their various combinations are evaluated and compared on images captured in a production environment, where results show up to 15% improvement using combinations of basic procedures. Reducing the simulation to reality gap in this way can aid to utilize the true potential of robot assisted production using artificial intelligence.