# From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection

arXiv: https://arxiv.org/abs/2609.38010 · HTML: https://mlsanity.com/paper/2609.38010/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Unity Perception: Generate Synthetic Data for Computer Vision](https://mlsanity.com/paper/2107.04259/) (2021)
- [Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks](https://mlsanity.com/paper/1612.07429/) (2016)
- [Deep SVBRDF Estimation on Real Materials](https://mlsanity.com/paper/2010.04143/) (2020)
- [On the Limitation of Diffusion Models for Synthesizing Training Datasets](https://mlsanity.com/paper/2311.13090/) (2023)
- [Self-Supervised YOLO: Leveraging Contrastive Learning for Label-Efficient Object Detection](https://mlsanity.com/paper/2508.01966/) (2025)

## Abstract (arXiv)

Modern computer vision models achieve high accuracy when trained on large-scale annotated datasets. In critical domains such as construction safety monitoring, data collection is costly, hazardous, and ethically constrained. This paper presents a systematic study comparing two complementary data generation paradigms, (1) Unity Simulation-based rendering and (2) Controllable Diffusion-based generation (CIA), for object detection under real data-scarce conditions. A unified experimental framework enables controlled dataset mixing across real, simulated, and generative sources, while maintaining identical model and training settings. Quantitative evaluation using Precision, Recall, mAP, and custom Delta-metrics, reveals that neither simulation nor generative augmentation alone achieves optimal transferability. Unity-only training yields an mAP@0.5 drop of -50% relative to real data, while CIA-only training shows a milder -16.5% degradation. Hybrid compositions significantly improve performance, with the 90% real + 10% Unity configuration achieving the best overall mAP@0.5 of 62.68% (+7.64% over baseline), and the 90% real + 10% CIA configuration maximizing precision at 74.45%. Results demonstrate that limited synthetic inclusion enhances generalization, while excessive substitution induces domain drift.