# A Survey on Datasets for Decision-making of Autonomous Vehicle

arXiv: https://arxiv.org/abs/2306.16784 · HTML: https://mlsanity.com/paper/2306.16784/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 95 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 30 of 45)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [One Million Scenes for Autonomous Driving: ONCE Dataset](https://mlsanity.com/paper/2106.11037/) (2021)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Self-Driving Cars: A Survey](https://mlsanity.com/paper/1901.04407/) (2019)
- [WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving](https://mlsanity.com/paper/1905.01489/) (2019)
- [V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting](https://mlsanity.com/paper/2305.05938/) (2023)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving](https://mlsanity.com/paper/1807.03776/) (2018)
- [Canadian Adverse Driving Conditions Dataset](https://mlsanity.com/paper/2001.10117/) (2020)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes](https://mlsanity.com/paper/1903.01568/) (2019)
- [Boreas: A Multi-Season Autonomous Driving Dataset](https://mlsanity.com/paper/2203.10168/) (2022)
- [SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving](https://mlsanity.com/paper/2005.03844/) (2020)
- [A*3D Dataset: Towards Autonomous Driving in Challenging Environments](https://mlsanity.com/paper/1909.07541/) (2019)
- [TITAN: Future Forecast using Action Priors](https://mlsanity.com/paper/2003.13886/) (2020)
- [DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction](https://mlsanity.com/paper/2005.08605/) (2020)
- [ROAD: The ROad event Awareness Dataset for Autonomous Driving](https://mlsanity.com/paper/2102.11585/) (2021)
- [Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles](https://mlsanity.com/paper/1910.04586/) (2019)
- [Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving](https://mlsanity.com/paper/2210.09539/) (2022)

## Built on by (most cited first, 2 of 2)

- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [Multimodal Data Storage and Retrieval for Embodied AI: A Survey](https://mlsanity.com/paper/2508.13901/) (2025)

## Abstract (arXiv)

Autonomous vehicles (AV) are expected to reshape future transportation systems, and decision-making is one of the critical modules toward high-level automated driving. To overcome those complicated scenarios that rule-based methods could not cope with well, data-driven decision-making approaches have aroused more and more focus. The datasets to be used in developing data-driven methods dramatically influences the performance of decision-making, hence it is necessary to have a comprehensive insight into the existing datasets. From the aspects of collection sources, driving data can be divided into vehicle, environment, and driver related data. This study compares the state-of-the-art datasets of these three categories and summarizes their features including sensors used, annotation, and driving scenarios. Based on the characteristics of the datasets, this survey also concludes the potential applications of datasets on various aspects of AV decision-making, assisting researchers to find appropriate ones to support their own research. The future trends of AV dataset development are summarized.