# From Real to Synthetic and Back: Synthesizing Training Data for Multi-Person Scene Understanding

arXiv: https://arxiv.org/abs/2006.02110 · HTML: https://mlsanity.com/paper/2006.02110/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2020.

## Built on by (most cited first, 3 of 3)

- [PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models](https://mlsanity.com/paper/2207.05025/) (2022)
- [PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision](https://mlsanity.com/paper/2112.09290/) (2021)
- [Do as we do: Multiple Person Video-To-Video Transfer](https://mlsanity.com/paper/2104.04721/) (2021)

## Abstract (arXiv)

We present a method for synthesizing naturally looking images of multiple people interacting in a specific scenario. These images benefit from the advantages of synthetic data: being fully controllable and fully annotated with any type of standard or custom-defined ground truth. To reduce the synthetic-to-real domain gap, we introduce a pipeline consisting of the following steps: 1) we render scenes in a context modeled after the real world, 2) we train a human parsing model on the synthetic images, 3) we use the model to estimate segmentation maps for real images, 4) we train a conditional generative adversarial network (cGAN) to learn the inverse mapping -- from a segmentation map to a real image, and 5) given new synthetic segmentation maps, we use the cGAN to generate realistic images. An illustration of our pipeline is presented in Figure 2. We use the generated data to train a multi-task model on the challenging tasks of UV mapping and dense depth estimation. We demonstrate the value of the data generation and the trained model, both quantitatively and qualitatively on the CMU Panoptic Dataset.