# Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

arXiv: https://arxiv.org/abs/1708.05869 · HTML: https://mlsanity.com/paper/1708.05869/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2017.

## Built on by (most cited first, 30 of 31)

- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021)
- [Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world](https://mlsanity.com/paper/2206.09889/) (2022)
- [GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS](https://mlsanity.com/paper/2408.01584/) (2024)
- [Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving](https://mlsanity.com/paper/2311.11056/) (2023)
- [MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility](https://mlsanity.com/paper/2407.08725/) (2024)
- [BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision](https://mlsanity.com/paper/2402.04519/) (2024)
- [OIL: Observational Imitation Learning](https://mlsanity.com/paper/1803.01129/) (2018)
- [Attribute Descent: Simulating Object-Centric Datasets on the Content Level and Beyond](https://mlsanity.com/paper/2202.14034/) (2022)
- [BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.10856/) (2026)
- [FusionPlanner: A Multi-task Motion Planner for Mining Trucks via Multi-sensor Fusion](https://mlsanity.com/paper/2308.06931/) (2023)
- [Synthetic Data-based Detection of Zebras in Drone Imagery](https://mlsanity.com/paper/2305.00432/) (2023)
- [Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic](https://mlsanity.com/paper/2310.04416/) (2023)
- [Adversarial and Reactive Traffic Entities for Behavior-Realistic Driving Simulation: A Review](https://mlsanity.com/paper/2409.14196/) (2024)
- [Getting SMARTER for Motion Planning in Autonomous Driving Systems](https://mlsanity.com/paper/2502.15824/) (2025)
- [UAV-MM3D: A Large-Scale Synthetic Benchmark for 3D Perception of Unmanned Aerial Vehicles with Multi-Modal Data](https://mlsanity.com/paper/2511.22404/) (2025)
- [Forest Inspection Dataset for Aerial Semantic Segmentation and Depth Estimation](https://mlsanity.com/paper/2403.06621/) (2024)
- [GCISG: Guided Causal Invariant Learning for Improved Syn-to-real Generalization](https://mlsanity.com/paper/2208.10024/) (2022)
- [Analyzing Effects of Fake Training Data on the Performance of Deep Learning Systems](https://mlsanity.com/paper/2303.01268/) (2023)
- [Learning from synthetic data generated with GRADE](https://mlsanity.com/paper/2305.04282/) (2023)
- [Image change detection with only a few samples](https://mlsanity.com/paper/2311.03762/) (2023)
- [SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation](https://mlsanity.com/paper/2311.16671/) (2023)
- [UAV-CROWD: Violent and non-violent crowd activity simulator from the perspective of UAV](https://mlsanity.com/paper/2208.06702/) (2022)
- [Vehicle-road Cooperative Simulation and 3D Visualization System](https://mlsanity.com/paper/2208.07304/) (2022)
- [WHU-Synthetic: A Synthetic Perception Dataset for 3-D Multitask Model Research](https://mlsanity.com/paper/2402.19059/) (2024)
- [Sky-Drive: A Distributed Multi-Agent Simulation Platform for Human-AI Collaborative and Socially-Aware Future Transportation](https://mlsanity.com/paper/2504.18010/) (2025)
- [VirtualEnv: A Platform for Embodied AI Research](https://mlsanity.com/paper/2601.07553/) (2026)
- [Diagnosing Aerial-View Object Detectors with Foundational Image Generative Models](https://mlsanity.com/paper/2607.02718/) (2026)

## Abstract (arXiv)

We present a photo-realistic training and evaluation simulator (Sim4CV) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network (DNN) architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool. The supplementary video can be viewed a https://youtu.be/SqAxzsQ7qUU