# Synthetic Datasets for Autonomous Driving: A Survey

arXiv: https://arxiv.org/abs/2304.12205 · HTML: https://mlsanity.com/paper/2304.12205/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 113 of its 174 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 30 of 113)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)

## Built on by (most cited first, 30 of 48)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation](https://mlsanity.com/paper/2512.12751/) (2025)
- [Deep Transfer Learning for Intelligent Vehicle Perception: a Survey](https://mlsanity.com/paper/2306.15110/) (2023)
- [Collaborative Perception Datasets for Autonomous Driving: A Review](https://mlsanity.com/paper/2504.12696/) (2025)
- [ACCIDENT: A Benchmark Dataset for Vehicle Accident Detection from Traffic Surveillance Videos](https://mlsanity.com/paper/2604.09819/) (2026)
- [PC-NeRF: Parent-Child Neural Radiance Fields Using Sparse LiDAR Frames in Autonomous Driving Environments](https://mlsanity.com/paper/2402.09325/) (2024)
- [Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2405.19202/) (2024)
- [Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models](https://mlsanity.com/paper/2405.20991/) (2024)
- [SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model](https://mlsanity.com/paper/2503.13952/) (2025)
- [AgentSGEN: Multi-Agent LLM in the Loop for Semantic Collaboration and GENeration of Synthetic Data](https://mlsanity.com/paper/2505.13466/) (2025)
- [PC-NeRF: Parent-Child Neural Radiance Fields under Partial Sensor Data Loss in Autonomous Driving Environments](https://mlsanity.com/paper/2310.00874/) (2023)
- [TK-Planes: Tiered K-Planes with High Dimensional Feature Vectors for Dynamic UAV-based Scenes](https://mlsanity.com/paper/2405.02762/) (2024)
- [IMFL-AIGC: Incentive Mechanism Design for Federated Learning Empowered by Artificial Intelligence Generated Content](https://mlsanity.com/paper/2406.08526/) (2024)
- [SynDiff-AD: Improving Semantic Segmentation and End-to-End Autonomous Driving with Synthetic Data from Latent Diffusion Models](https://mlsanity.com/paper/2411.16776/) (2024)
- [Large Language Models for Crash Detection in Video: A Survey of Methods, Datasets, and Challenges](https://mlsanity.com/paper/2507.02074/) (2025)
- [Scaling Up Forest Vision with Synthetic Data](https://mlsanity.com/paper/2509.11201/) (2025)
- [Synthetic Dataset Evaluation Based on Generalized Cross Validation](https://mlsanity.com/paper/2509.11273/) (2025)
- [LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents](https://mlsanity.com/paper/2512.17445/) (2025)
- [A Sustainable AI Economy Needs Data Deals That Work for Generators](https://mlsanity.com/paper/2601.09966/) (2026)
- [A Style-Based Profiling Framework for Quantifying the Synthetic-to-Real Gap in Autonomous Driving Datasets](https://mlsanity.com/paper/2510.10203/) (2025)
- [DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding](https://mlsanity.com/paper/2511.02495/) (2025)
- [SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Dataset](https://mlsanity.com/paper/2502.01894/) (2025)
- [Evaluating the Impact of Synthetic Data on Object Detection Tasks in Autonomous Driving](https://mlsanity.com/paper/2503.09803/) (2025)
- [Synthetic Dataset Generation for Autonomous Mobile Robots Using 3D Gaussian Splatting for Vision Training](https://mlsanity.com/paper/2506.05092/) (2025)
- [David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation](https://mlsanity.com/paper/2510.14115/) (2025)
- [On the Assessment of Sensitivity of Autonomous Vehicle Perception](https://mlsanity.com/paper/2602.00314/) (2026)
- [Utilizing dataset affinity prediction in object detection to assess training data](https://mlsanity.com/paper/2311.09768/) (2023)
- [A Unified Generative Framework for Realistic Lidar Simulation in Autonomous Driving Systems](https://mlsanity.com/paper/2312.15817/) (2023)

## Abstract (arXiv)

Autonomous driving techniques have been flourishing in recent years while thirsting for huge amounts of high-quality data. However, it is difficult for real-world datasets to keep up with the pace of changing requirements due to their expensive and time-consuming experimental and labeling costs. Therefore, more and more researchers are turning to synthetic datasets to easily generate rich and changeable data as an effective complement to the real world and to improve the performance of algorithms. In this paper, we summarize the evolution of synthetic dataset generation methods and review the work to date in synthetic datasets related to single and multi-task categories for to autonomous driving study. We also discuss the role that synthetic dataset plays the evaluation, gap test, and positive effect in autonomous driving related algorithm testing, especially on trustworthiness and safety aspects. Finally, we discuss general trends and possible development directions. To the best of our knowledge, this is the first survey focusing on the application of synthetic datasets in autonomous driving. This survey also raises awareness of the problems of real-world deployment of autonomous driving technology and provides researchers with a possible solution.