# SURE-Val: Safe Urban Relevance Extension and Validation

arXiv: https://arxiv.org/abs/2308.02266 · HTML: https://mlsanity.com/paper/2308.02266/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [One Million Scenes for Autonomous Driving: ONCE Dataset](https://mlsanity.com/paper/2106.11037/) (2021)
- [Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations](https://mlsanity.com/paper/2008.05930/) (2020)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](https://mlsanity.com/paper/2106.15004/) (2021)
- [Learning to Evaluate Perception Models Using Planner-Centric Metrics](https://mlsanity.com/paper/2004.08745/) (2020)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Tasks](https://mlsanity.com/paper/2001.08001/) (2020)
- [Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Models](https://mlsanity.com/paper/2204.02392/) (2022)
- [Revisiting 3D Object Detection From an Egocentric Perspective](https://mlsanity.com/paper/2112.07787/) (2021)
- [A Review of Testing Object-Based Environment Perception for Safe Automated Driving](https://mlsanity.com/paper/2102.08460/) (2021)
- [Zero-shot Deep Reinforcement Learning Driving Policy Transfer for Autonomous Vehicles based on Robust Control](https://mlsanity.com/paper/1812.03216/) (2018)
- [Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation](https://mlsanity.com/paper/2206.12471/) (2022)
- [Watch Out for the Safety-Threatening Actors: Proactively Mitigating Safety Hazards](https://mlsanity.com/paper/2206.00886/) (2022)

## Built on by (most cited first, 3 of 3)

- [Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving](https://mlsanity.com/paper/2512.15181/) (2025)
- [Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Driving](https://mlsanity.com/paper/2308.07106/) (2023)
- [Scenario-Independent Criticality Assessment and Prediction for Vulnerable Road Users in Autonomous Driving](https://mlsanity.com/paper/2609.11947/) (2026)

## Abstract (arXiv)

To evaluate perception components of an automated driving system, it is necessary to define the relevant objects. While the urban domain is popular among perception datasets, relevance is insufficiently specified for this domain. Therefore, this work adopts an existing method to define relevance in the highway domain and expands it to the urban domain. While different conceptualizations and definitions of relevance are present in literature, there is a lack of methods to validate these definitions. Therefore, this work presents a novel relevance validation method leveraging a motion prediction component. The validation leverages the idea that removing irrelevant objects should not influence a prediction component which reflects human driving behavior. The influence on the prediction is quantified by considering the statistical distribution of prediction performance across a large-scale dataset. The validation procedure is verified using criteria specifically designed to exclude relevant objects. The validation method is successfully applied to the relevance criteria from this work, thus supporting their validity.