mlsanityData to 14 Oct 2026

A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook

arXiv 2401.01454 ↗First posted 2 Jan 2024cs.CVcited by 66

Not on the map. Why?

Cited by 20 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autonomous driving has rapidly developed and shown promising performance due to recent advances in hardware and deep learning techniques. High-quality datasets are fundamental for developing reliable autonomous driving algorithms. Previous dataset surveys either focused on a limited number or lacked detailed investigation of dataset characteristics. To this end, we present an exhaustive study of 265 autonomous driving datasets from multiple perspectives, including sensor modalities, data size, tasks, and contextual conditions. We introduce a novel metric to evaluate the impact of datasets, which can also be a guide for creating new datasets. Besides, we analyze the annotation processes, existing labeling tools, and the annotation quality of datasets, showing the importance of establishing a standard annotation pipeline. On the other hand, we thoroughly analyze the impact of geographical and adversarial environmental conditions on the performance of autonomous driving systems. Moreover, we exhibit the data distribution of several vital datasets and discuss their pros and cons accordingly. Finally, we discuss the current challenges and the development trend of the future autonomous driving datasets.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 217 of its 360 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

217 of 360 references.

References · 217

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  6. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  7. 2022Planning-oriented Autonomous Drivingcited by 1,087
  8. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  9. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  10. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  12. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
38 more
  1. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  2. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  4. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
  5. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  6. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  7. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  8. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  9. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 439
  10. 2017Semantic Foggy Scene Understanding with Synthetic Datacited by 403
  11. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 403
  12. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  13. 2020Virtual KITTI 2cited by 353
  14. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  15. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 341
  16. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  17. 2022V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformercited by 320
  18. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  19. 2021OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communicationcited by 304
  20. 2021DSEC: A Stereo Event Camera Dataset for Driving Scenarioscited by 303
  21. 2020V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Predictioncited by 292
  22. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  23. 2021HDMapNet: An Online HD Map Construction and Evaluation Frameworkcited by 281
  24. 2018The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systemscited by 277
  25. 2022DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detectioncited by 273
  26. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  27. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  28. 2016A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehiclescited by 254
  29. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 240
  30. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  31. 2018Unity: A General Platform for Intelligent Agentscited by 236
  32. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  33. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  34. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 231
  35. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  36. 2023VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Trackingcited by 222
  37. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  38. 2018The Unmanned Aerial Vehicle Benchmark: Object Detection and Trackingcited by 208

50 most cited of 217 All 217 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 35; 2026: 18.

66 papers.

Sort:

By area:

  1. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 176
  2. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
  3. 20253D and 4D World Modeling: A Surveycited by 67
  4. 2024MAN TruckScenes: A multimodal dataset for autonomous trucking in diverse conditionscited by 41
  5. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  6. 2024OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Drivingcited by 27
  7. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  8. 2024ActiveAnno3D -- An Active Learning Framework for Multi-Modal 3D Object Detectioncited by 17
  9. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMscited by 11
  10. 2024XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesiscited by 9
  11. 2024WARM-3D: A Weakly-Supervised Sim2Real Domain Adaptation Framework for Roadside Monocular 3D Object Detectioncited by 9
  12. 2024LOGen: Toward Lidar Object Generation by Point Diffusioncited by 9
38 more
  1. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  2. 2024COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Drivingcited by 8
  3. 2025Collaborative Perception Datasets for Autonomous Driving: A Reviewcited by 8
  4. 2025TrajDiff: End-to-end Autonomous Driving without Perception Annotationcited by 8
  5. 2024SUSTechGAN: Image Generation for Object Detection in Adverse Conditions of Autonomous Drivingcited by 5
  6. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  7. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  8. 2025Continual Adaptation for Autonomous Driving with the Mixture of Progressive Experts Networkcited by 4
  9. 2025Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretrainingcited by 4
  10. 2025Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Drivingcited by 4
  11. 2024GraphRelate3D: Context-Dependent 3D Object Detection with Inter-Object Relationship Graphscited by 3
  12. 2024SynDiff-AD: Improving Semantic Segmentation and End-to-End Autonomous Driving with Synthetic Data from Latent Diffusion Modelscited by 3
  13. 2025L2RDaS: Synthesizing 4D Radar Tensors for Model Generalization via Dataset Expansioncited by 3
  14. 2025Benchmarking the Spatial Robustness of DNNs via Natural and Adversarial Localized Corruptionscited by 3
  15. 2025Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selectioncited by 3
  16. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  17. 2026123D: Unifying Multi-Modal Autonomous Driving Data at Scalecited by 3
  18. 2024PointCompress3D: A Point Cloud Compression Framework for Roadside LiDARs in Intelligent Transportation Systemscited by 2
  19. 2025Towards Vision Zero: The TUM Traffic Accid3nD Datasetcited by 2
  20. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 2
  21. 2025Visual Heading Prediction for Autonomous Aerial Vehiclescited by 2
  22. 2025Enhancing Highway Safety: Accident Detection on the A9 Test Stretch Using Roadside Sensorscited by 1
  23. 2025SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Datasetcited by 1
  24. 2025What Demands Attention in Urban Street Scenes? From Scene Understanding towards Road Safety: A Survey of Vision-driven Datasets and Studiescited by 1
  25. 2025AutoVDC: Automated Vision Data Cleaning Using Vision-Language Modelscited by 1
  26. 2025Temporally Consistent Unsupervised Segmentation for Mobile Robot Perceptioncited by 1
  27. 2025Safety-Critical Learning for Long-Tail Events: The TUM Traffic Accident Datasetcited by 1
  28. 20252COOOL: 2nd Workshop on the Challenge Of Out-Of-Label Hazards in Autonomous Drivingcited by 1
  29. 2025GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regionscited by 1
  30. 2025VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusioncited by 1
  31. 2026HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Trafficcited by 1
  32. 2026ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarioscited by 1
  33. 2026Scaling-Aware Data Selection for End-to-End Autonomous Driving Systemscited by 1
  34. 2026CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction in Challenging Road Topographycited by 1
  35. 20264D Radar Meets LiDAR and Camera: Cooperative Perception under Adverse Weathercited by 1
  36. 2025I2V-GS: Infrastructure-to-Vehicle View Transformation with Gaussian Splatting for Autonomous Driving Data Generationcited by 0
  37. 2025OpenConstruction: A Systematic Synthesis of Open Visual Datasets for Data-Centric Artificial Intelligence in Construction Monitoringcited by 0
  38. 2025MLPerf Automotivecited by 0

50 most cited of 66 All 66 (JSON)