# Object Detector Differences when using Synthetic and Real Training Data

arXiv: https://arxiv.org/abs/2312.00694 · HTML: https://mlsanity.com/paper/2312.00694/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Fixing the train-test resolution discrepancy](https://mlsanity.com/paper/1906.06423/) (2019)
- [Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?](https://mlsanity.com/paper/1610.01983/) (2016)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [What shapes feature representations? Exploring datasets, architectures, and training](https://mlsanity.com/paper/2006.12433/) (2020)
- [Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing](https://mlsanity.com/paper/1810.08705/) (2018)
- [Are All Layers Created Equal?](https://mlsanity.com/paper/1902.01996/) (2019)
- [On Pre-Trained Image Features and Synthetic Images for Deep Learning](https://mlsanity.com/paper/1710.10710/) (2017)
- [A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts](https://mlsanity.com/paper/2105.00290/) (2021)
- [How much real data do we actually need: Analyzing object detection performance using synthetic and real data](https://mlsanity.com/paper/1907.07061/) (2019)

## Built on by (most cited first, 5 of 5)

- [Generative AI-based Pipeline Architecture for Increasing Training Efficiency in Intelligent Weed Control Systems](https://mlsanity.com/paper/2411.00548/) (2024)
- [Investigation of the Impact of Synthetic Training Data in the Industrial Application of Terminal Strip Object Detection](https://mlsanity.com/paper/2403.04809/) (2024)
- [Addressing the Depth-of-Field Constraint: A New Paradigm for High Resolution Multi-Focus Image Fusion](https://mlsanity.com/paper/2510.19581/) (2025)
- [Object Detection for Autonomous Driving in Chinese Rural Scenes: An Experimental Study on Real-Synthetic Data Mixing and Model Evaluation](https://mlsanity.com/paper/2607.27058/) (2026)
- [Domain generalization and synthetic data in object detection: the enabler, the probe, and the gap](https://mlsanity.com/paper/2610.00030/) (2026)

## Abstract (arXiv)

To train well-performing generalizing neural networks, sufficiently large and diverse datasets are needed. Collecting data while adhering to privacy legislation becomes increasingly difficult and annotating these large datasets is both a resource-heavy and time-consuming task. An approach to overcome these difficulties is to use synthetic data since it is inherently scalable and can be automatically annotated. However, how training on synthetic data affects the layers of a neural network is still unclear. In this paper, we train the YOLOv3 object detector on real and synthetic images from city environments. We perform a similarity analysis using Centered Kernel Alignment (CKA) to explore the effects of training on synthetic data on a layer-wise basis. The analysis captures the architecture of the detector while showing both different and similar patterns between different models. With this similarity analysis we want to give insights on how training synthetic data affects each layer and to give a better understanding of the inner workings of complex neural networks. The results show that the largest similarity between a detector trained on real data and a detector trained on synthetic data was in the early layers, and the largest difference was in the head part. The results also show that no major difference in performance or similarity could be seen between frozen and unfrozen backbone.