# Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Driving

arXiv: https://arxiv.org/abs/2308.07106 · HTML: https://mlsanity.com/paper/2308.07106/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 21 of 21)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections](https://mlsanity.com/paper/1911.07602/) (2019)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [Learning to Evaluate Perception Models Using Planner-Centric Metrics](https://mlsanity.com/paper/2004.08745/) (2020)
- [3D Multi-Object Tracking: A Baseline and New Evaluation Metrics](https://mlsanity.com/paper/1907.03961/) (2019)
- [A Survey on Automated Driving System Testing: Landscapes and Trends](https://mlsanity.com/paper/2206.05961/) (2022)
- [Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Tasks](https://mlsanity.com/paper/2001.08001/) (2020)
- [6-Layer Model for a Structured Description and Categorization of Urban Traffic and Environment](https://mlsanity.com/paper/2012.06319/) (2020)
- [Injecting Planning-Awareness into Prediction and Detection Evaluation](https://mlsanity.com/paper/2110.03270/) (2021)
- [How Trustworthy are Performance Evaluations for Basic Vision Tasks?](https://mlsanity.com/paper/2008.03533/) (2020)
- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)
- [Inferring Spatial Uncertainty in Object Detection](https://mlsanity.com/paper/2003.03644/) (2020)
- [Risk Ranked Recall: Collision Safety Metric for Object Detection Systems in Autonomous Vehicles](https://mlsanity.com/paper/2106.04146/) (2021)
- [A Delay Metric for Video Object Detection: What Average Precision Fails to Tell](https://mlsanity.com/paper/1908.06368/) (2019)
- [A Review of Testing Object-Based Environment Perception for Safe Automated Driving](https://mlsanity.com/paper/2102.08460/) (2021)
- [Transcendental Idealism of Planner: Evaluating Perception from Planning Perspective for Autonomous Driving](https://mlsanity.com/paper/2306.07276/) (2023)
- [Evaluating Object (mis)Detection from a Safety and Reliability Perspective: Discussion and Measures](https://mlsanity.com/paper/2203.02205/) (2022)
- [On the validation of complex systems operating in open contexts](https://mlsanity.com/paper/1902.10517/) (2019)
- [SURE-Val: Safe Urban Relevance Extension and Validation](https://mlsanity.com/paper/2308.02266/) (2023)
- [MTBF Model for AVs -- From Perception Errors to Vehicle-Level Failures](https://mlsanity.com/paper/2205.02621/) (2022)

## Abstract (arXiv)

The object perception of automated driving systems must pass quality and robustness tests before a safe deployment. Such tests typically identify true positive (TP), false-positive (FP), and false-negative (FN) detections and aggregate them to metrics. Since the literature seems to be lacking a comprehensive way to define the identification of TPs/FPs/FNs, this paper provides a checklist of relevant functional aspects and implementation details. Besides labeling policies of the test set, we cover areas of vision, occlusion handling, safety-relevant areas, matching criteria, temporal and probabilistic issues, and further aspects. Even though the checklist cannot be fully formalized, it can help practitioners minimize the ambiguity of their tests, which, in turn, makes statements on object perception more reliable and comparable.