# Creating Impactful Autonomous Driving Datasets: A Strategic Guide from Research Gap to Benchmark

arXiv: https://arxiv.org/abs/2607.00710 · HTML: https://mlsanity.com/paper/2607.00710/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 48 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 30 of 38)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2208.14437/) (2022)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [One Million Scenes for Autonomous Driving: ONCE Dataset](https://mlsanity.com/paper/2106.11037/) (2021)
- [MapTRv2: An End-to-End Framework for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2308.05736/) (2023)
- [The ApolloScape Open Dataset for Autonomous Driving and its Application](https://mlsanity.com/paper/1803.06184/) (2018)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving](https://mlsanity.com/paper/2112.12610/) (2021)
- [StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Construction](https://mlsanity.com/paper/2308.12570/) (2023)
- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models](https://mlsanity.com/paper/2506.09042/) (2025)
- [OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping](https://mlsanity.com/paper/2304.10440/) (2023)
- [Zenseact Open Dataset: A large-scale and diverse multimodal dataset for autonomous driving](https://mlsanity.com/paper/2305.02008/) (2023)
- [MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping](https://mlsanity.com/paper/2403.15951/) (2024)
- [NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving](https://mlsanity.com/paper/2404.07762/) (2024)
- [Stream Query Denoising for Vectorized HD Map Construction](https://mlsanity.com/paper/2401.09112/) (2024)
- [A Commute in Data: The comma2k19 Dataset](https://mlsanity.com/paper/1812.05752/) (2018)
- [Enhancing Vectorized Map Perception with Historical Rasterized Maps](https://mlsanity.com/paper/2409.00620/) (2024)
- [MapExpert: Online HD Map Construction with Simple and Efficient Sparse Map Element Expert](https://mlsanity.com/paper/2412.12704/) (2024)

## Abstract (arXiv)

Well-designed autonomous driving datasets have fundamentally shaped research progress, yet existing literature primarily describes what datasets contain rather than how to strategically design impactful ones. This is especially limiting for small and medium-sized labs and startups that cannot afford to misallocate scarce resources. We argue that impactful dataset creation begins with a diagnosis: whether a research question is blocked by a data problem or an evaluation problem, and proceeds by selecting the minimal data operator(s) that closes the resulting gap, recording new data only when no cheaper operator(s) suffices. We analyze the evolution of major autonomous driving (AD) datasets through this lens and distill a strategic framework spanning gap identification, operator choice, sensor suite design, and annotation strategy. We ground the framework in a running case study of our KITScenes dataset family. The datasets are available at: https://kitscenes.com/