# AI Playground: Unreal Engine-based Data Ablation Tool for Deep Learning

arXiv: https://arxiv.org/abs/2007.06153 · HTML: https://mlsanity.com/paper/2007.06153/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 5 of 5)

- [A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?](https://mlsanity.com/paper/2108.11018/) (2021)
- [Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond](https://mlsanity.com/paper/2309.00064/) (2023)
- [SuperCaustics: Real-time, open-source simulation of transparent objects for deep learning applications](https://mlsanity.com/paper/2107.11008/) (2021)
- [Fully Automated Tree Topology Estimation and Artery-Vein Classification](https://mlsanity.com/paper/2202.02382/) (2022)
- [Domain-Transferred Synthetic Data Generation for Improving Monocular Depth Estimation](https://mlsanity.com/paper/2405.01113/) (2024)

## Abstract (arXiv)

Machine learning requires data, but acquiring and labeling real-world data is challenging, expensive, and time-consuming. More importantly, it is nearly impossible to alter real data post-acquisition (e.g., change the illumination of a room), making it very difficult to measure how specific properties of the data affect performance. In this paper, we present AI Playground (AIP), an open-source, Unreal Engine-based tool for generating and labeling virtual image data. With AIP, it is trivial to capture the same image under different conditions (e.g., fidelity, lighting, etc.) and with different ground truths (e.g., depth or surface normal values). AIP is easily extendable and can be used with or without code. To validate our proposed tool, we generated eight datasets of otherwise identical but varying lighting and fidelity conditions. We then trained deep neural networks to predict (1) depth values, (2) surface normals, or (3) object labels and assessed each network's intra- and cross-dataset performance. Among other insights, we verified that sensitivity to different settings is problem-dependent. We confirmed the findings of other studies that segmentation models are very sensitive to fidelity, but we also found that they are just as sensitive to lighting. In contrast, depth and normal estimation models seem to be less sensitive to fidelity or lighting and more sensitive to the structure of the image. Finally, we tested our trained depth-estimation networks on two real-world datasets and obtained results comparable to training on real data alone, confirming that our virtual environments are realistic enough for real-world tasks.