# Testing Rare Downstream Safety Violations via Upstream Adaptive Sampling of Perception Error Models

arXiv: https://arxiv.org/abs/2209.09674 · HTML: https://mlsanity.com/paper/2209.09674/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation](https://mlsanity.com/paper/1811.00145/) (2018)
- [Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques](https://mlsanity.com/paper/1605.04965/) (2016)
- [Neural Bridge Sampling for Evaluating Safety-Critical Autonomous Systems](https://mlsanity.com/paper/2008.10581/) (2020)
- [Compositional Falsification of Cyber-Physical Systems with Machine Learning Components](https://mlsanity.com/paper/1703.00978/) (2017)
- [Efficient Black-box Assessment of Autonomous Vehicle Safety](https://mlsanity.com/paper/1912.03618/) (2019)
- [Arithmetic-Geometric Mean Robustness for Control from Signal Temporal Logic Specifications](https://mlsanity.com/paper/1903.05186/) (2019)
- [Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing](https://mlsanity.com/paper/2003.02381/) (2020)
- [How Do We Fail? Stress Testing Perception in Autonomous Vehicles](https://mlsanity.com/paper/2203.14155/) (2022)
- [Control from Signal Temporal Logic Specifications with Smooth Cumulative Quantitative Semantics](https://mlsanity.com/paper/1904.11611/) (2019)
- [Semantic Adversarial Deep Learning](https://mlsanity.com/paper/1804.07045/) (2018)
- [Statistical Verification of Autonomous Systems using Surrogate Models and Conformal Inference](https://mlsanity.com/paper/2004.00279/) (2020)
- [Scalable Autonomous Vehicle Safety Validation through Dynamic Programming and Scene Decomposition](https://mlsanity.com/paper/2004.06801/) (2020)
- [A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation](https://mlsanity.com/paper/2110.02739/) (2021)

## Built on by (most cited first, 7 of 7)

- [PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles](https://mlsanity.com/paper/2302.11919/) (2023)
- [Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Rendering](https://mlsanity.com/paper/2309.15770/) (2023)
- [Attacking Motion Planners Using Adversarial Perception Errors](https://mlsanity.com/paper/2311.12722/) (2023)
- [EMPERROR: A Flexible Generative Perception Error Model for Probing Self-Driving Planners](https://mlsanity.com/paper/2411.07719/) (2024)
- [Adaptive Splitting of Reusable Temporal Monitors for Rare Traffic Violations](https://mlsanity.com/paper/2405.15771/) (2024)
- [Control Analysis and Design for Autonomous Vehicles Subject to Imperfect AI-Based Perception](https://mlsanity.com/paper/2509.12137/) (2025)
- [Approaches to Analysis and Design of AI-Based Autonomous Vehicles](https://mlsanity.com/paper/2509.12169/) (2025)

## Abstract (arXiv)

Testing black-box perceptual-control systems in simulation faces two difficulties. Firstly, perceptual inputs in simulation lack the fidelity of real-world sensor inputs. Secondly, for a reasonably accurate perception system, encountering a rare failure trajectory may require running infeasibly many simulations. This paper combines perception error models -- surrogates for a sensor-based detection system -- with state-dependent adaptive importance sampling. This allows us to efficiently assess the rare failure probabilities for real-world perceptual control systems within simulation. Our experiments with an autonomous braking system equipped with an RGB obstacle-detector show that our method can calculate accurate failure probabilities with an inexpensive number of simulations. Further, we show how choice of safety metric can influence the process of learning proposal distributions capable of reliably sampling high-probability failures.