# Diffusion Dataset Generation: Towards Closing the Sim2Real Gap for Pedestrian Detection

arXiv: https://arxiv.org/abs/2305.09401 · HTML: https://mlsanity.com/paper/2305.09401/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 16 May 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 16 of 16)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [CityPersons: A Diverse Dataset for Pedestrian Detection](https://mlsanity.com/paper/1702.05693/) (2017)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World](https://mlsanity.com/paper/1803.08319/) (2018)
- [MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?](https://mlsanity.com/paper/2108.09518/) (2021)
- [Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images](https://mlsanity.com/paper/2301.04802/) (2023)
- [Robust pedestrian detection in thermal imagery using synthesized images](https://mlsanity.com/paper/2102.02005/) (2021)
- [Expecting the Unexpected: Training Detectors for Unusual Pedestrians with Adversarial Imposters](https://mlsanity.com/paper/1703.06283/) (2017)
- [Attention-based Adversarial Appearance Learning of Augmented Pedestrians](https://mlsanity.com/paper/2107.02673/) (2021)

## Built on by (most cited first, 3 of 3)

- [POCI-Diff: 3D-Layout Guided Diffusion for Controllable Synthetic Surveillance Data Generation](https://mlsanity.com/paper/2601.14056/) (2026)
- [DALI-PD: Diffusion-based Synthetic Layout Heatmap Generation for ML in Physical Design](https://mlsanity.com/paper/2507.10606/) (2025)
- [From Camera-Based Sensing to Reasoning: A Comprehensive Review Toward Proactive Vulnerable Road User Safety](https://mlsanity.com/paper/2510.03314/) (2025)

## Abstract (arXiv)

We propose a method that augments a simulated dataset using diffusion models to improve the performance of pedestrian detection in real-world data. The high cost of collecting and annotating data in the real-world has motivated the use of simulation platforms to create training datasets. While simulated data is inexpensive to collect and annotate, it unfortunately does not always closely match the distribution of real-world data, which is known as the sim2real gap. In this paper we propose a novel method of synthetic data creation meant to close the sim2real gap for the challenging pedestrian detection task. Our method uses a diffusion-based architecture to learn a real-world distribution which, once trained, is used to generate datasets. We mix this generated data with simulated data as a form of augmentation and show that training on a combination of generated and simulated data increases average precision by as much as 27.3% for pedestrian detection models in real-world data, compared against training on purely simulated data.