mlsanityData to 7 Oct 2026

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

arXiv 2102.08460 ↗First posted 16 Feb 2021cs.RO cs.CV cs.SYcited by 9

Not on the map. Why?

Abstract (arXiv)

Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. We focus on testing for verification and validation purposes at the interface between perception and planning, and structure our analysis along the three axes 1) test criteria and metrics, 2) test scenarios, and 3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. We find that the realization of safety-aware perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 41 of its 181 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

41 of 181 references.

References · 41

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  4. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  5. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  6. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  7. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  8. 2018Efficient Formal Safety Analysis of Neural Networkscited by 165
  9. 2017On a Formal Model of Safe and Scalable Self-driving Carscited by 129
  10. 2018Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detectioncited by 82
  11. 2020Learning to Evaluate Perception Models Using Planner-Centric Metricscited by 49
  12. 2018Scenarios for Development, Test and Validation of Automated Vehiclescited by 45
29 more
  1. 2017Ontology based Scene Creation for the Development of Automated Vehiclescited by 40
  2. 2018Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industrycited by 38
  3. 2016Towards Verified Artificial Intelligencecited by 36
  4. 2016Generalized optimal sub-pattern assignment metriccited by 33
  5. 2020Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Taskscited by 31
  6. 2019Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Drivingcited by 30
  7. 2017An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Softwarecited by 26
  8. 20206-Layer Model for a Structured Description and Categorization of Urban Traffic and Environmentcited by 26
  9. 2020A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applicationscited by 19
  10. 2019Providentia -- A Large-Scale Sensor System for the Assistance of Autonomous Vehicles and Its Evaluationcited by 17
  11. 2016Extended Object Tracking: Introduction, Overview and Applicationscited by 16
  12. 2020Fundamental Considerations around Scenario-Based Testing for Automated Drivingcited by 15
  13. 2018Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262cited by 13
  14. 2019Towards a Framework to Manage Perceptual Uncertainty for Safe Automated Drivingcited by 12
  15. 2018Weak in the NEES?: Auto-tuning Kalman Filters with Bayesian Optimizationcited by 10
  16. 2020Inferring Spatial Uncertainty in Object Detectioncited by 10
  17. 2020The efficacy of Neural Planning Metrics: A meta-analysis of PKL on nuScenescited by 7
  18. 2020Monitoring and Diagnosability of Perception Systemscited by 7
  19. 2017Towards a Functional System Architecture for Automated Vehiclescited by 6
  20. 2019On the validation of complex systems operating in open contextscited by 6
  21. 2019From Functional to Logical Scenarios: Detailing a Keyword-Based Scenario Description for Execution in a Simulation Environmentcited by 5
  22. 2020Vehicle Position Estimation with Aerial Imagery from Unmanned Aerial Vehiclescited by 3
  23. 2020Supporting Safe Decision Making Through Holistic System-Level Representations & Monitoring -- A Summary and Taxonomy of Self-Representation Concepts for Automated Vehiclescited by 2
  24. 2019Automated Ground Truth Estimation of Vulnerable Road Users in Automotive Radar Data Using GNSScited by 1
  25. 2020Extraction and Assessment of Naturalistic Human Driving Trajectories from Infrastructure Camera and Radar Sensorscited by 1
  26. 2020Benchmarking LiDAR Sensors for Development and Evaluation of Automotive Perceptioncited by 1
  27. 2020Reducing Uncertainty by Fusing Dynamic Occupancy Grid Maps in a Cloud-based Collective Environment Modelcited by 1
  28. 2020High-Precision Digital Traffic Recording with Multi-LiDAR Infrastructure Sensor Setupscited by 1
  29. 2020Sensors, Safety Models and A System-Level Approach to Safe and Scalable Automated Vehiclescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 0; 2025: 4; 2026: 1.

9 papers.

  1. 2025Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Testcited by 18
  2. 2023Dialogue-based generation of self-driving simulation scenarios using Large Language Modelscited by 7
  3. 2023PEM: Perception Error Model for Virtual Testing of Autonomous Vehiclescited by 4
  4. 2023SURE-Val: Safe Urban Relevance Extension and Validationcited by 3
  5. 2025CODEI: Resource-Efficient Task-Driven Co-Design of Perception and Decision Making for Mobile Robots Applied to Autonomous Vehiclescited by 1
  6. 2025Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Datacited by 1
  7. 2023Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Drivingcited by 0
  8. 2025A Systematic Literature Review on Safety of the Intended Functionality for Automated Driving Systemscited by 0
  9. 2026Data Leakage in Automotive Perception: Practitioners' Insightscited by 0