# Play and Learn: Using Video Games to Train Computer Vision Models

arXiv: https://arxiv.org/abs/1608.01745 · HTML: https://mlsanity.com/paper/1608.01745/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2016.

## Built on by (most cited first, 16 of 16)

- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [CAD2Render: A Modular Toolkit for GPU-accelerated Photorealistic Synthetic Data Generation for the Manufacturing Industry](https://mlsanity.com/paper/2211.14054/) (2022)
- [Universal pre-training by iterated random computation](https://mlsanity.com/paper/2506.20057/) (2025)
- [SIMPL: Generating Synthetic Overhead Imagery to Address Zero-shot and Few-Shot Detection Problems](https://mlsanity.com/paper/2106.15681/) (2021)
- [Image Segmentation to Identify Safe Landing Zones for Unmanned Aerial Vehicles](https://mlsanity.com/paper/2111.14557/) (2021)
- [Label-Free Synthetic Pretraining of Object Detectors](https://mlsanity.com/paper/2208.04268/) (2022)
- [Behavioural Cloning in VizDoom](https://mlsanity.com/paper/2401.03993/) (2024)
- [End-To-End Training and Testing Gamification Framework to Learn Human Highway Driving](https://mlsanity.com/paper/2404.10849/) (2024)
- [SYNOSIS: Image synthesis pipeline for machine vision in metal surface inspection](https://mlsanity.com/paper/2410.14844/) (2024)
- [Lacking Data? No worries! How synthetic images can alleviate image scarcity in wildlife surveys: a case study with muskox (Ovibos moschatus)](https://mlsanity.com/paper/2511.11882/) (2025)
- [Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions](https://mlsanity.com/paper/2208.12763/) (2022)
- [Meta-simulation for the Automated Design of Synthetic Overhead Imagery](https://mlsanity.com/paper/2209.08685/) (2022)
- [Computer vision tasks for intelligent aerospace missions: An overview](https://mlsanity.com/paper/2407.06513/) (2024)
- [MC-hands-1M: A glove-wearing hand dataset for pose estimation](https://mlsanity.com/paper/2210.10428/) (2022)
- [ForestSim: A Synthetic Benchmark for Intelligent Vehicle Perception in Unstructured Forest Environments](https://mlsanity.com/paper/2603.27923/) (2026)

## Abstract (arXiv)

Video games are a compelling source of annotated data as they can readily provide fine-grained groundtruth for diverse tasks. However, it is not clear whether the synthetically generated data has enough resemblance to the real-world images to improve the performance of computer vision models in practice. We present experiments assessing the effectiveness on real-world data of systems trained on synthetic RGB images that are extracted from a video game. We collected over 60000 synthetic samples from a modern video game with similar conditions to the real-world CamVid and Cityscapes datasets. We provide several experiments to demonstrate that the synthetically generated RGB images can be used to improve the performance of deep neural networks on both image segmentation and depth estimation. These results show that a convolutional network trained on synthetic data achieves a similar test error to a network that is trained on real-world data for dense image classification. Furthermore, the synthetically generated RGB images can provide similar or better results compared to the real-world datasets if a simple domain adaptation technique is applied. Our results suggest that collaboration with game developers for an accessible interface to gather data is potentially a fruitful direction for future work in computer vision.