# Evaluating Object (mis)Detection from a Safety and Reliability Perspective: Discussion and Measures

arXiv: https://arxiv.org/abs/2203.02205 · HTML: https://mlsanity.com/paper/2203.02205/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [A General Framework for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/1907.06890/) (2019)
- [A Survey of Deep Learning-based Object Detection](https://mlsanity.com/paper/1907.09408/) (2019)
- [Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection](https://mlsanity.com/paper/1804.05132/) (2018)
- [SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds](https://mlsanity.com/paper/2004.02774/) (2020)
- [On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products](https://mlsanity.com/paper/1610.01256/) (2016)
- [Risk Ranked Recall: Collision Safety Metric for Object Detection Systems in Autonomous Vehicles](https://mlsanity.com/paper/2106.04146/) (2021)
- [Safety-Aware Hardening of 3D Object Detection Neural Network Systems](https://mlsanity.com/paper/2003.11242/) (2020)

## Built on by (most cited first, 8 of 8)

- [A Flow-based Credibility Metric for Safety-critical Pedestrian Detection](https://mlsanity.com/paper/2402.07642/) (2024)
- [Object criticality for safer navigation](https://mlsanity.com/paper/2406.10232/) (2024)
- [Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Driving](https://mlsanity.com/paper/2308.07106/) (2023)
- [Dynamic Classification: Leveraging Self-Supervised Classification to Enhance Prediction Performance](https://mlsanity.com/paper/2502.18891/) (2025)
- [Rethink 3D Object Detection from Physical World](https://mlsanity.com/paper/2507.00190/) (2025)
- [RoAD Benchmark: How LiDAR Models Fail under Coupled Domain Shifts and Label Evolution](https://mlsanity.com/paper/2601.07855/) (2026)
- [Effort-Based Criticality Metrics for Evaluating 3D Perception Errors in Autonomous Driving](https://mlsanity.com/paper/2603.28029/) (2026)
- [Detecting Backdoors in Object Detection via Pre-NMS Prediction Distribution Shift](https://mlsanity.com/paper/2608.19088/) (2026)

## Abstract (arXiv)

We argue that object detectors in the safety critical domain should prioritize detection of objects that are most likely to interfere with the actions of the autonomous actor. Especially, this applies to objects that can impact the actor's safety and reliability. To quantify the impact of object (mis)detection on safety and reliability in the context of autonomous driving, we propose new object detection measures that reward the correct identification of objects that are most dangerous and most likely to affect driving decisions. To achieve this, we build an object criticality model to reward the detection of the objects based on proximity, orientation, and relative velocity with respect to the subject vehicle. Then, we apply our model on the recent autonomous driving dataset nuScenes, and we compare nine object detectors. Results show that, in several settings, object detectors that perform best according to the nuScenes ranking are not the preferable ones when the focus is shifted on safety and reliability.