# Unity Perception: Generate Synthetic Data for Computer Vision

arXiv: https://arxiv.org/abs/2107.04259 · HTML: https://mlsanity.com/paper/2107.04259/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes](https://mlsanity.com/paper/2012.02924/) (2020)
- [BlenderProc](https://mlsanity.com/paper/1911.01911/) (2019)
- [ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors](https://mlsanity.com/paper/1904.03239/) (2019)
- [Domain Randomization and Generative Models for Robotic Grasping](https://mlsanity.com/paper/1710.06425/) (2017)
- [NViSII: A Scriptable Tool for Photorealistic Image Generation](https://mlsanity.com/paper/2105.13962/) (2021)
- [A Hierarchical Grocery Store Image Dataset with Visual and Semantic Labels](https://mlsanity.com/paper/1901.00711/) (2019)
- [An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detection](https://mlsanity.com/paper/1902.09967/) (2019)

## Built on by (most cited first, 30 of 43)

- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting](https://mlsanity.com/paper/2310.17323/) (2023)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models](https://mlsanity.com/paper/2207.05025/) (2022)
- [Synthetic Data Generation with LLM for Improved Depression Prediction](https://mlsanity.com/paper/2411.17672/) (2024)
- [PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision](https://mlsanity.com/paper/2112.09290/) (2021)
- [Knock, knock. Who's there? -- Identifying football player jersey numbers with synthetic data](https://mlsanity.com/paper/2203.00734/) (2022)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)
- [M3Act: Learning from Synthetic Human Group Activities](https://mlsanity.com/paper/2306.16772/) (2023)
- [WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance on Wheelchair Users](https://mlsanity.com/paper/2404.17063/) (2024)
- [LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding](https://mlsanity.com/paper/2508.19204/) (2025)
- [Enhancing Object Detection with Hybrid dataset in Manufacturing Environments: Comparing Federated Learning to Conventional Techniques](https://mlsanity.com/paper/2408.08974/) (2024)
- [Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation](https://mlsanity.com/paper/2504.08473/) (2025)
- [Automatic Generation of Synthetic Colonoscopy Videos for Domain Randomization](https://mlsanity.com/paper/2205.10368/) (2022)
- [STUPD: A Synthetic Dataset for Spatial and Temporal Relation Reasoning](https://mlsanity.com/paper/2309.06680/) (2023)
- [On the Equivalency, Substitutability, and Flexibility of Synthetic Data](https://mlsanity.com/paper/2403.16244/) (2024)
- [Supervised Representation Learning towards Generalizable Assembly State Recognition](https://mlsanity.com/paper/2408.11700/) (2024)
- [Improving Pallet Detection Using Synthetic Data](https://mlsanity.com/paper/2402.07098/) (2024)
- [Synthetic imagery for fuzzy object detection: A comparative study](https://mlsanity.com/paper/2410.01124/) (2024)
- [DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding](https://mlsanity.com/paper/2511.02495/) (2025)
- [MINERVAS: Massive INterior EnviRonments VirtuAl Synthesis](https://mlsanity.com/paper/2107.06149/) (2021)
- [Comparison of synthetic dataset generation methods for medical intervention rooms using medical clothing detection as an example](https://mlsanity.com/paper/2209.11493/) (2022)
- [AnthroNet: Conditional Generation of Humans via Anthropometrics](https://mlsanity.com/paper/2309.03812/) (2023)
- [Enhancing Object Detection Performance for Small Objects through Synthetic Data Generation and Proportional Class-Balancing Technique: A Comparative Study in Industrial Scenarios](https://mlsanity.com/paper/2401.12729/) (2024)
- [Using Game Engines and Machine Learning to Create Synthetic Satellite Imagery for a Tabletop Verification Exercise](https://mlsanity.com/paper/2404.11461/) (2024)
- [FAFA: Frequency-Aware Flow-Aided Self-Supervision for Underwater Object Pose Estimation](https://mlsanity.com/paper/2409.16600/) (2024)
- [Enhancing Object Detection Accuracy in Autonomous Vehicles Using Synthetic Data](https://mlsanity.com/paper/2411.15602/) (2024)
- [CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models](https://mlsanity.com/paper/2508.21732/) (2025)

## Abstract (arXiv)

We introduce the Unity Perception package which aims to simplify and accelerate the process of generating synthetic datasets for computer vision tasks by offering an easy-to-use and highly customizable toolset. This open-source package extends the Unity Editor and engine components to generate perfectly annotated examples for several common computer vision tasks. Additionally, it offers an extensible Randomization framework that lets the user quickly construct and configure randomized simulation parameters in order to introduce variation into the generated datasets. We provide an overview of the provided tools and how they work, and demonstrate the value of the generated synthetic datasets by training a 2D object detection model. The model trained with mostly synthetic data outperforms the model trained using only real data.