# Self-Supervised Real-to-Sim Scene Generation

arXiv: https://arxiv.org/abs/2011.14488 · HTML: https://mlsanity.com/paper/2011.14488/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 13 of 13)

- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects](https://mlsanity.com/paper/2204.03139/) (2022)
- [Phone2Proc: Bringing Robust Robots Into Our Chaotic World](https://mlsanity.com/paper/2212.04819/) (2022)
- [Optimizing Data Collection for Machine Learning](https://mlsanity.com/paper/2210.01234/) (2022)
- [Unpaired Image Translation via Vector Symbolic Architectures](https://mlsanity.com/paper/2209.02686/) (2022)
- [A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation](https://mlsanity.com/paper/2112.05068/) (2021)
- [Feature Generation for Long-tail Classification](https://mlsanity.com/paper/2111.05956/) (2021)
- [Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and Reweighting](https://mlsanity.com/paper/2302.04832/) (2023)
- [From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving](https://mlsanity.com/paper/2305.13960/) (2023)
- [Photo-realistic Neural Domain Randomization](https://mlsanity.com/paper/2210.12682/) (2022)
- [Label-Free Synthetic Pretraining of Object Detectors](https://mlsanity.com/paper/2208.04268/) (2022)
- [TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments](https://mlsanity.com/paper/2208.07943/) (2022)
- [Realistically distributing object placements in synthetic training data improves the performance of vision-based object detection models](https://mlsanity.com/paper/2305.14621/) (2023)

## Abstract (arXiv)

Synthetic data is emerging as a promising solution to the scalability issue of supervised deep learning, especially when real data are difficult to acquire or hard to annotate. Synthetic data generation, however, can itself be prohibitively expensive when domain experts have to manually and painstakingly oversee the process. Moreover, neural networks trained on synthetic data often do not perform well on real data because of the domain gap. To solve these challenges, we propose Sim2SG, a self-supervised automatic scene generation technique for matching the distribution of real data. Importantly, Sim2SG does not require supervision from the real-world dataset, thus making it applicable in situations for which such annotations are difficult to obtain. Sim2SG is designed to bridge both the content and appearance gaps, by matching the content of real data, and by matching the features in the source and target domains. We select scene graph (SG) generation as the downstream task, due to the limited availability of labeled datasets. Experiments demonstrate significant improvements over leading baselines in reducing the domain gap both qualitatively and quantitatively, on several synthetic datasets as well as the real-world KITTI dataset.