# Neural-Sim: Learning to Generate Training Data with NeRF

arXiv: https://arxiv.org/abs/2207.11368 · HTML: https://mlsanity.com/paper/2207.11368/

## Key facts

- Cited by 27 papers in the map (data through 7 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Builds on (most cited first, 30 of 35)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [On the "steerability" of generative adversarial networks](https://mlsanity.com/paper/1907.07171/) (2019)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [CodeNeRF: Disentangled Neural Radiance Fields for Object Categories](https://mlsanity.com/paper/2109.01750/) (2021)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols](https://mlsanity.com/paper/1502.03143/) (2015)
- [Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation](https://mlsanity.com/paper/1804.06534/) (2018)
- [BlenderProc](https://mlsanity.com/paper/1911.01911/) (2019)
- [Sim2real transfer learning for 3D human pose estimation: motion to the rescue](https://mlsanity.com/paper/1907.02499/) (2019)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Photorealistic Image Synthesis for Object Instance Detection](https://mlsanity.com/paper/1902.03334/) (2019)
- [Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation](https://mlsanity.com/paper/2008.09092/) (2020)
- [Learning To Simulate](https://mlsanity.com/paper/1810.02513/) (2018)
- [Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks](https://mlsanity.com/paper/1612.07429/) (2016)

## Built on by (most cited first, 27 of 27)

- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [Instructional Fingerprinting of Large Language Models](https://mlsanity.com/paper/2401.12255/) (2024)
- [Expanding Small-Scale Datasets with Guided Imagination](https://mlsanity.com/paper/2211.13976/) (2022)
- [NeRF-Supervised Deep Stereo](https://mlsanity.com/paper/2303.17603/) (2023)
- [Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting](https://mlsanity.com/paper/2403.09413/) (2024)
- [3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D Detection](https://mlsanity.com/paper/2312.05277/) (2023)
- [Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?](https://mlsanity.com/paper/2212.10784/) (2022)
- [AI-Generated Images as Data Source: The Dawn of Synthetic Era](https://mlsanity.com/paper/2310.01830/) (2023)
- [BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation](https://mlsanity.com/paper/2405.09546/) (2024)
- [SAR-NeRF: Neural Radiance Fields for Synthetic Aperture Radar Multi-View Representation](https://mlsanity.com/paper/2307.05087/) (2023)
- [Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use cases](https://mlsanity.com/paper/2308.07118/) (2023)
- [Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation](https://mlsanity.com/paper/2309.05956/) (2023)
- [DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation](https://mlsanity.com/paper/2312.14216/) (2023)
- [Domain Generalization for 6D Pose Estimation Through NeRF-based Image Synthesis](https://mlsanity.com/paper/2407.10762/) (2024)
- [SonarSplat: Novel View Synthesis of Imaging Sonar via Gaussian Splatting](https://mlsanity.com/paper/2504.00159/) (2025)
- [EM-Paste: EM-guided Cut-Paste with DALL-E Augmentation for Image-level Weakly Supervised Instance Segmentation](https://mlsanity.com/paper/2212.07629/) (2022)
- [Dual-Camera Smooth Zoom on Mobile Phones](https://mlsanity.com/paper/2404.04908/) (2024)
- [Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation](https://mlsanity.com/paper/2504.08473/) (2025)
- [TK-Planes: Tiered K-Planes with High Dimensional Feature Vectors for Dynamic UAV-based Scenes](https://mlsanity.com/paper/2405.02762/) (2024)
- [NeRF-Enhanced Outpainting for Faithful Field-of-View Extrapolation](https://mlsanity.com/paper/2309.13240/) (2023)
- [MetaSTNet: Multimodal Meta-learning for Cellular Traffic Conformal Prediction](https://mlsanity.com/paper/2505.21553/) (2025)
- [High-Quality Real-Time Rendering Using Subpixel Sampling Reconstruction](https://mlsanity.com/paper/2301.01036/) (2023)
- [Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methods](https://mlsanity.com/paper/2403.08352/) (2024)
- [Leveraging the Third Dimension in Contrastive Learning](https://mlsanity.com/paper/2301.11790/) (2023)
- [MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generation](https://mlsanity.com/paper/2603.19575/) (2026)
- [EventHub: Data Factory for Generalizable Event-Based Stereo Networks without Active Sensors](https://mlsanity.com/paper/2604.02331/) (2026)
- [CAD-Free Learning of Spacecraft Pose Estimators via NeRF-Based Augmentations](https://mlsanity.com/paper/2605.19649/) (2026)

## Abstract (arXiv)

Training computer vision models usually requires collecting and labeling vast amounts of imagery under a diverse set of scene configurations and properties. This process is incredibly time-consuming, and it is challenging to ensure that the captured data distribution maps well to the target domain of an application scenario. Recently, synthetic data has emerged as a way to address both of these issues. However, existing approaches either require human experts to manually tune each scene property or use automatic methods that provide little to no control; this requires rendering large amounts of random data variations, which is slow and is often suboptimal for the target domain. We present the first fully differentiable synthetic data pipeline that uses Neural Radiance Fields (NeRFs) in a closed-loop with a target application's loss function. Our approach generates data on-demand, with no human labor, to maximize accuracy for a target task. We illustrate the effectiveness of our method on synthetic and real-world object detection tasks. We also introduce a new "YCB-in-the-Wild" dataset and benchmark that provides a test scenario for object detection with varied poses in real-world environments.