# WorldGen: A Large Scale Generative Simulator

arXiv: https://arxiv.org/abs/2210.00715 · HTML: https://mlsanity.com/paper/2210.00715/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 63 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 30 of 43)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering](https://mlsanity.com/paper/2112.05504/) (2021)
- [Deep Bilateral Learning for Real-Time Image Enhancement](https://mlsanity.com/paper/1707.02880/) (2017)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [AutoFlow: Learning a Better Training Set for Optical Flow](https://mlsanity.com/paper/2104.14544/) (2021)
- [SceneGen: Learning to Generate Realistic Traffic Scenes](https://mlsanity.com/paper/2101.06541/) (2021)
- [BlenderProc](https://mlsanity.com/paper/1911.01911/) (2019)
- [Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras](https://mlsanity.com/paper/2203.08896/) (2022)
- [Learning to Extract a Video Sequence from a Single Motion-Blurred Image](https://mlsanity.com/paper/1804.04065/) (2018)
- [PASS: An ImageNet replacement for self-supervised pretraining without humans](https://mlsanity.com/paper/2109.13228/) (2021)
- [EV-IMO: Motion Segmentation Dataset and Learning Pipeline for Event Cameras](https://mlsanity.com/paper/1903.07520/) (2019)
- [EVDodgeNet: Deep Dynamic Obstacle Dodging with Event Cameras](https://mlsanity.com/paper/1906.02919/) (2019)
- [Single Image Depth Estimation Trained via Depth from Defocus Cues](https://mlsanity.com/paper/2001.05036/) (2020)
- [UnrealCV: Connecting Computer Vision to Unreal Engine](https://mlsanity.com/paper/1609.01326/) (2016)
- [SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic](https://mlsanity.com/paper/1911.04074/) (2019)

## Built on by (most cited first, 4 of 4)

- [Microsaccade-inspired Event Camera for Robotics](https://mlsanity.com/paper/2405.17769/) (2024)
- [Learning autonomous driving from aerial imagery](https://mlsanity.com/paper/2410.14177/) (2024)
- [SKoPe3D: A Synthetic Dataset for Vehicle Keypoint Perception in 3D from Traffic Monitoring Cameras](https://mlsanity.com/paper/2309.01324/) (2023)
- [MinNav: Minimalist Navigation Using Optical Flow For Active Tiny Aerial Robots](https://mlsanity.com/paper/2606.07813/) (2026)

## Abstract (arXiv)

In the era of deep learning, data is the critical determining factor in the performance of neural network models. Generating large datasets suffers from various difficulties such as scalability, cost efficiency and photorealism. To avoid expensive and strenuous dataset collection and annotations, researchers have inclined towards computer-generated datasets. Although, a lack of photorealism and a limited amount of computer-aided data, has bounded the accuracy of network predictions. To this end, we present WorldGen -- an open source framework to autonomously generate countless structured and unstructured 3D photorealistic scenes such as city view, object collection, and object fragmentation along with its rich ground truth annotation data. WorldGen being a generative model gives the user full access and control to features such as texture, object structure, motion, camera and lens properties for better generalizability by diminishing the data bias in the network. We demonstrate the effectiveness of WorldGen by presenting an evaluation on deep optical flow. We hope such a tool can open doors for future research in a myriad of domains related to robotics and computer vision by reducing manual labor and the cost of acquiring rich and high-quality data.