# A Review of Testing Object-Based Environment Perception for Safe Automated Driving

arXiv: https://arxiv.org/abs/2102.08460 · HTML: https://mlsanity.com/paper/2102.08460/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 181 references that are in the map.
- First posted on arXiv 16 Feb 2021.

## Builds on (most cited first, 30 of 41)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Efficient Formal Safety Analysis of Neural Networks](https://mlsanity.com/paper/1809.08098/) (2018)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection](https://mlsanity.com/paper/1804.05132/) (2018)
- [Learning to Evaluate Perception Models Using Planner-Centric Metrics](https://mlsanity.com/paper/2004.08745/) (2020)
- [Scenarios for Development, Test and Validation of Automated Vehicles](https://mlsanity.com/paper/1801.08598/) (2018)
- [Ontology based Scene Creation for the Development of Automated Vehicles](https://mlsanity.com/paper/1704.01006/) (2017)
- [Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry](https://mlsanity.com/paper/1812.05389/) (2018)
- [Towards Verified Artificial Intelligence](https://mlsanity.com/paper/1606.08514/) (2016)
- [Generalized optimal sub-pattern assignment metric](https://mlsanity.com/paper/1601.05585/) (2016)
- [Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Tasks](https://mlsanity.com/paper/2001.08001/) (2020)
- [Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Driving](https://mlsanity.com/paper/1909.12358/) (2019)
- [An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software](https://mlsanity.com/paper/1709.02435/) (2017)
- [6-Layer Model for a Structured Description and Categorization of Urban Traffic and Environment](https://mlsanity.com/paper/2012.06319/) (2020)
- [A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications](https://mlsanity.com/paper/2006.15172/) (2020)
- [Providentia -- A Large-Scale Sensor System for the Assistance of Autonomous Vehicles and Its Evaluation](https://mlsanity.com/paper/1906.06789/) (2019)
- [Extended Object Tracking: Introduction, Overview and Applications](https://mlsanity.com/paper/1604.00970/) (2016)
- [Fundamental Considerations around Scenario-Based Testing for Automated Driving](https://mlsanity.com/paper/2005.04045/) (2020)
- [Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262](https://mlsanity.com/paper/1808.01614/) (2018)
- [Towards a Framework to Manage Perceptual Uncertainty for Safe Automated Driving](https://mlsanity.com/paper/1903.03438/) (2019)
- [Weak in the NEES?: Auto-tuning Kalman Filters with Bayesian Optimization](https://mlsanity.com/paper/1807.08855/) (2018)
- [Inferring Spatial Uncertainty in Object Detection](https://mlsanity.com/paper/2003.03644/) (2020)
- [The efficacy of Neural Planning Metrics: A meta-analysis of PKL on nuScenes](https://mlsanity.com/paper/2010.09350/) (2020)
- [Monitoring and Diagnosability of Perception Systems](https://mlsanity.com/paper/2011.07010/) (2020)

## Built on by (most cited first, 9 of 9)

- [Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test](https://mlsanity.com/paper/2503.02911/) (2025)
- [Dialogue-based generation of self-driving simulation scenarios using Large Language Models](https://mlsanity.com/paper/2310.17372/) (2023)
- [PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles](https://mlsanity.com/paper/2302.11919/) (2023)
- [SURE-Val: Safe Urban Relevance Extension and Validation](https://mlsanity.com/paper/2308.02266/) (2023)
- [CODEI: Resource-Efficient Task-Driven Co-Design of Perception and Decision Making for Mobile Robots Applied to Autonomous Vehicles](https://mlsanity.com/paper/2503.10296/) (2025)
- [Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data](https://mlsanity.com/paper/2511.02994/) (2025)
- [Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Driving](https://mlsanity.com/paper/2308.07106/) (2023)
- [A Systematic Literature Review on Safety of the Intended Functionality for Automated Driving Systems](https://mlsanity.com/paper/2503.02498/) (2025)
- [Data Leakage in Automotive Perception: Practitioners' Insights](https://mlsanity.com/paper/2604.06899/) (2026)

## Abstract (arXiv)

Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. We focus on testing for verification and validation purposes at the interface between perception and planning, and structure our analysis along the three axes 1) test criteria and metrics, 2) test scenarios, and 3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. We find that the realization of safety-aware perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.