# UnrealCV: Connecting Computer Vision to Unreal Engine

arXiv: https://arxiv.org/abs/1609.01326 · HTML: https://mlsanity.com/paper/1609.01326/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2016.

## Built on by (most cited first, 30 of 41)

- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation](https://mlsanity.com/paper/2412.00596/) (2024)
- [OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2111.14341/) (2021)
- [Ego-Only: Egocentric Action Detection without Exocentric Transferring](https://mlsanity.com/paper/2301.01380/) (2023)
- [SCAMPS: Synthetics for Camera Measurement of Physiological Signals](https://mlsanity.com/paper/2206.04197/) (2022)
- [UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene](https://mlsanity.com/paper/2310.13263/) (2023)
- [NViSII: A Scriptable Tool for Photorealistic Image Generation](https://mlsanity.com/paper/2105.13962/) (2021)
- [Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles](https://mlsanity.com/paper/2112.12252/) (2021)
- [Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose](https://mlsanity.com/paper/2110.14213/) (2021)
- [SPADES: A Realistic Spacecraft Pose Estimation Dataset using Event Sensing](https://mlsanity.com/paper/2311.05310/) (2023)
- [OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2304.10266/) (2023)
- [Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing](https://mlsanity.com/paper/2110.04902/) (2021)
- [Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processing](https://mlsanity.com/paper/2402.03082/) (2024)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [Image-based monitoring of bolt loosening through deep-learning-based integrated detection and tracking](https://mlsanity.com/paper/2111.09117/) (2021)
- [Differential Evolution based Dual Adversarial Camouflage: Fooling Human Eyes and Object Detectors](https://mlsanity.com/paper/2210.08870/) (2022)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)
- [Object Detection and Pose Estimation from RGB and Depth Data for Real-time, Adaptive Robotic Grasping](https://mlsanity.com/paper/2101.07347/) (2021)
- [WorldGen: A Large Scale Generative Simulator](https://mlsanity.com/paper/2210.00715/) (2022)
- [DT/MARS-CycleGAN: Improved Object Detection for MARS Phenotyping Robot](https://mlsanity.com/paper/2310.12787/) (2023)
- [UnrealROX+: An Improved Tool for Acquiring Synthetic Data from Virtual 3D Environments](https://mlsanity.com/paper/2104.11776/) (2021)
- [Learning to compose 6-DoF omnidirectional videos using multi-sphere images](https://mlsanity.com/paper/2103.05842/) (2021)
- [TAGPerson: A Target-Aware Generation Pipeline for Person Re-identification](https://mlsanity.com/paper/2112.14239/) (2021)
- [Synthetic Data for Object Classification in Industrial Applications](https://mlsanity.com/paper/2212.04790/) (2022)
- [Improving Pallet Detection Using Synthetic Data](https://mlsanity.com/paper/2402.07098/) (2024)
- [A Hybrid Co-Finetuning Approach for Visual Bug Detection in Video Games](https://mlsanity.com/paper/2510.03591/) (2025)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations](https://mlsanity.com/paper/2608.15659/) (2026)
- [UAV Obstacle Avoidance by Human-in-the-Loop Reinforcement in Arbitrary 3D Environment](https://mlsanity.com/paper/2304.05959/) (2023)
- [SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes](https://mlsanity.com/paper/2307.07333/) (2023)

## Abstract (arXiv)

Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (http://unrealcv.github.io) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms.