# Towards Scenario- and Capability-Driven Dataset Development and Evaluation: An Approach in the Context of Mapless Automated Driving

arXiv: https://arxiv.org/abs/2404.19656 · HTML: https://mlsanity.com/paper/2404.19656/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [The ApolloScape Open Dataset for Autonomous Driving and its Application](https://mlsanity.com/paper/1803.06184/) (2018)
- [Spatial As Deep: Spatial CNN for Traffic Scene Understanding](https://mlsanity.com/paper/1712.06080/) (2017)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [CurveLane-NAS: Unifying Lane-Sensitive Architecture Search and Adaptive Point Blending](https://mlsanity.com/paper/2007.12147/) (2020)
- [Gen-LaneNet: A Generalized and Scalable Approach for 3D Lane Detection](https://mlsanity.com/paper/2003.10656/) (2020)
- [Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?](https://mlsanity.com/paper/1912.08320/) (2019)
- [VPGNet: Vanishing Point Guided Network for Lane and Road Marking Detection and Recognition](https://mlsanity.com/paper/1710.06288/) (2017)
- [Requirements Engineering for Machine Learning: Perspectives from Data Scientists](https://mlsanity.com/paper/1908.04674/) (2019)
- [Scenarios for Development, Test and Validation of Automated Vehicles](https://mlsanity.com/paper/1801.08598/) (2018)
- [Real time Detection of Lane Markers in Urban Streets](https://mlsanity.com/paper/1411.7113/) (2014)
- [Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks](https://mlsanity.com/paper/1903.02193/) (2019)
- [Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future](https://mlsanity.com/paper/2312.03408/) (2023)
- [VIL-100: A New Dataset and A Baseline Model for Video Instance Lane Detection](https://mlsanity.com/paper/2108.08482/) (2021)
- [Ad-datasets: a meta-collection of data sets for autonomous driving](https://mlsanity.com/paper/2202.01909/) (2022)
- [Robust Lane Detection via Expanded Self Attention](https://mlsanity.com/paper/2102.07037/) (2021)
- [Towards a Skill- And Ability-Based Development Process for Self-Aware Automated Road Vehicles](https://mlsanity.com/paper/1708.02532/) (2017)
- [Ego-Lane Analysis System (ELAS): Dataset and Algorithms](https://mlsanity.com/paper/1806.05984/) (2018)
- [A Dataset for Lane Instance Segmentation in Urban Environments](https://mlsanity.com/paper/1807.01347/) (2018)
- [CeyMo: See More on Roads -- A Novel Benchmark Dataset for Road Marking Detection](https://mlsanity.com/paper/2110.11867/) (2021)
- [A System's Perspective Towards an Architecture Framework for Safe Automated Vehicles](https://mlsanity.com/paper/1804.07020/) (2018)
- [A Knowledge-based Approach for the Automatic Construction of Skill Graphs for Online Monitoring](https://mlsanity.com/paper/2102.08827/) (2021)
- [The Grind for Good Data: Understanding ML Practitioners' Struggles and Aspirations in Making Good Data](https://mlsanity.com/paper/2211.14981/) (2022)

## Abstract (arXiv)

The foundational role of datasets in defining the capabilities of deep learning models has led to their rapid proliferation. At the same time, published research focusing on the process of dataset development for environment perception in automated driving has been scarce, thereby reducing the applicability of openly available datasets and impeding the development of effective environment perception systems. Sensor-based, mapless automated driving is one of the contexts where this limitation is evident. While leveraging real-time sensor data, instead of pre-defined HD maps promises enhanced adaptability and safety by effectively navigating unexpected environmental changes, it also increases the demands on the scope and complexity of the information provided by the perception system. To address these challenges, we propose a scenario- and capability-based approach to dataset development. Grounded in the principles of ISO 21448 (safety of the intended functionality, SOTIF), extended by ISO/TR 4804, our approach facilitates the structured derivation of dataset requirements. This not only aids in the development of meaningful new datasets but also enables the effective comparison of existing ones. Applying this methodology to a broad range of existing lane detection datasets, we identify significant limitations in current datasets, particularly in terms of real-world applicability, a lack of labeling of critical features, and an absence of comprehensive information for complex driving maneuvers.