# Meta-simulation for the Automated Design of Synthetic Overhead Imagery

arXiv: https://arxiv.org/abs/2209.08685 · HTML: https://mlsanity.com/paper/2209.08685/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 60 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes](https://mlsanity.com/paper/1708.01566/) (2017)
- [Deep Convolutional Inverse Graphics Network](https://mlsanity.com/paper/1503.03167/) (2015)
- [Building Generalizable Agents with a Realistic and Rich 3D Environment](https://mlsanity.com/paper/1801.02209/) (2018)
- [Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing](https://mlsanity.com/paper/1810.08705/) (2018)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation](https://mlsanity.com/paper/2008.09092/) (2020)
- [Auto-Tuned Sim-to-Real Transfer](https://mlsanity.com/paper/2104.07662/) (2021)
- [RarePlanes: Synthetic Data Takes Flight](https://mlsanity.com/paper/2006.02963/) (2020)
- [Learning To Simulate](https://mlsanity.com/paper/1810.02513/) (2018)
- [Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks](https://mlsanity.com/paper/1612.07429/) (2016)

## Built on by (most cited first, 1 of 1)

- [Grounding Stylistic Domain Generalization with Quantitative Domain Shift Measures and Synthetic Scene Images](https://mlsanity.com/paper/2405.15961/) (2024)

## Abstract (arXiv)

The use of synthetic (or simulated) data for training machine learning models has grown rapidly in recent years. Synthetic data can often be generated much faster and more cheaply than its real-world counterpart. One challenge of using synthetic imagery however is scene design: e.g., the choice of content and its features and spatial arrangement. To be effective, this design must not only be realistic, but appropriate for the target domain, which (by assumption) is unlabeled. In this work, we propose an approach to automatically choose the design of synthetic imagery based upon unlabeled real-world imagery. Our approach, termed Neural-Adjoint Meta-Simulation (NAMS), builds upon the seminal recent meta-simulation approaches. In contrast to the current state-of-the-art methods, our approach can be pre-trained once offline, and then provides fast design inference for new target imagery. Using both synthetic and real-world problems, we show that NAMS infers synthetic designs that match both the in-domain and out-of-domain target imagery, and that training segmentation models with NAMS-designed imagery yields superior results compared to na"ive randomized designs and state-of-the-art meta-simulation methods.