# AutoSimulate: (Quickly) Learning Synthetic Data Generation

arXiv: https://arxiv.org/abs/2008.08424 · HTML: https://mlsanity.com/paper/2008.08424/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2020.

## Built on by (most cited first, 11 of 11)

- [3DB: A Framework for Debugging Computer Vision Models](https://mlsanity.com/paper/2106.03805/) (2021)
- [Neural-Sim: Learning to Generate Training Data with NeRF](https://mlsanity.com/paper/2207.11368/) (2022)
- [Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data](https://mlsanity.com/paper/2112.00054/) (2021)
- [A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation](https://mlsanity.com/paper/2303.09165/) (2023)
- [DIVA: Dataset Derivative of a Learning Task](https://mlsanity.com/paper/2111.09785/) (2021)
- [Human Body Measurement Estimation with Adversarial Augmentation](https://mlsanity.com/paper/2210.05667/) (2022)
- [Meta-simulation for the Automated Design of Synthetic Overhead Imagery](https://mlsanity.com/paper/2209.08685/) (2022)
- [AutoSynth: Learning to Generate 3D Training Data for Object Point Cloud Registration](https://mlsanity.com/paper/2309.11170/) (2023)
- [Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methods](https://mlsanity.com/paper/2403.08352/) (2024)
- [Learning Instance-Specific Parameters of Black-Box Models Using Differentiable Surrogates](https://mlsanity.com/paper/2407.17530/) (2024)
- [Bringing Balance to Hand Shape Classification: Mitigating Data Imbalance Through Generative Models](https://mlsanity.com/paper/2507.17008/) (2025)

## Abstract (arXiv)

Simulation is increasingly being used for generating large labelled datasets in many machine learning problems. Recent methods have focused on adjusting simulator parameters with the goal of maximising accuracy on a validation task, usually relying on REINFORCE-like gradient estimators. However these approaches are very expensive as they treat the entire data generation, model training, and validation pipeline as a black-box and require multiple costly objective evaluations at each iteration. We propose an efficient alternative for optimal synthetic data generation, based on a novel differentiable approximation of the objective. This allows us to optimize the simulator, which may be non-differentiable, requiring only one objective evaluation at each iteration with a little overhead. We demonstrate on a state-of-the-art photorealistic renderer that the proposed method finds the optimal data distribution faster (up to 50×), with significantly reduced training data generation (up to 30×) and better accuracy (+8.7%) on real-world test datasets than previous methods.