# TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments

arXiv: https://arxiv.org/abs/2208.07943 · HTML: https://mlsanity.com/paper/2208.07943/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 16 Aug 2022.

## Builds on (most cited first, 30 of 31)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2111.14887/) (2021)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments](https://mlsanity.com/paper/1811.10200/) (2018)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes](https://mlsanity.com/paper/1708.01566/) (2017)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing](https://mlsanity.com/paper/1810.08705/) (2018)
- [BlenderProc](https://mlsanity.com/paper/1911.01911/) (2019)
- [OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets](https://mlsanity.com/paper/2007.12868/) (2020)
- [STEP: Segmenting and Tracking Every Pixel](https://mlsanity.com/paper/2102.11859/) (2021)
- [Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation](https://mlsanity.com/paper/2008.09092/) (2020)
- [What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?](https://mlsanity.com/paper/1801.06397/) (2018)
- [Learning To Simulate](https://mlsanity.com/paper/1810.02513/) (2018)
- [SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic](https://mlsanity.com/paper/1911.04074/) (2019)
- [Cityscapes 3D: Dataset and Benchmark for 9 DoF Vehicle Detection](https://mlsanity.com/paper/2006.07864/) (2020)
- [Contrastive Syn-to-Real Generalization](https://mlsanity.com/paper/2104.02290/) (2021)
- [How much real data do we actually need: Analyzing object detection performance using synthetic and real data](https://mlsanity.com/paper/1907.07061/) (2019)

## Built on by (most cited first, 2 of 2)

- [MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection](https://mlsanity.com/paper/2504.06801/) (2025)
- [Exploring the Impact of Synthetic Data for Aerial-view Human Detection](https://mlsanity.com/paper/2405.15203/) (2024)

## Abstract (arXiv)

High-quality structured data with rich annotations are critical components in intelligent vehicle systems dealing with road scenes. However, data curation and annotation require intensive investments and yield low-diversity scenarios. The recently growing interest in synthetic data raises questions about the scope of improvement in such systems and the amount of manual work still required to produce high volumes and variations of simulated data. This work proposes a synthetic data generation pipeline that utilizes existing datasets, like nuScenes, to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation, mimicking real scene properties with high-fidelity, along with mechanisms to diversify samples in a physically meaningful way. We demonstrate improvements in mIoU metrics by presenting qualitative and quantitative experiments with real and synthetic data for semantic segmentation on the Cityscapes and KITTI-STEP datasets. All relevant code and data is released on github (https://github.com/shubham1810/trove_toolkit).