# Bayesian Safety Validation for Failure Probability Estimation of Black-Box Systems

arXiv: https://arxiv.org/abs/2305.02449 · HTML: https://mlsanity.com/paper/2305.02449/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 52 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 10 of 10)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Algorithms for Verifying Deep Neural Networks](https://mlsanity.com/paper/1903.06758/) (2019)
- [A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization](https://mlsanity.com/paper/2111.05040/) (2021)
- [A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems](https://mlsanity.com/paper/2005.02979/) (2020)
- [OVERT: An Algorithm for Safety Verification of Neural Network Control Policies for Nonlinear Systems](https://mlsanity.com/paper/2108.01220/) (2021)
- [A Survey of Monte Carlo Methods for Parameter Estimation](https://mlsanity.com/paper/2107.11820/) (2021)
- [GaussianProcesses.jl: A Nonparametric Bayes package for the Julia Language](https://mlsanity.com/paper/1812.09064/) (2018)
- [Cross-Entropy Method Variants for Optimization](https://mlsanity.com/paper/2009.09043/) (2020)
- [Sequential Bayesian experimental design for estimation of extreme-event probability in stochastic dynamical systems](https://mlsanity.com/paper/2102.11108/) (2021)

## Built on by (most cited first, 5 of 5)

- [Predictive Uncertainty for Runtime Assurance of a Real-Time Computer Vision-Based Landing System](https://mlsanity.com/paper/2508.09732/) (2025)
- [Formal and Practical Elements for the Certification of Machine Learning Systems](https://mlsanity.com/paper/2310.03217/) (2023)
- [Gaussian process surrogate model to approximate power grid simulators -- An application to the certification of a congestion management controller](https://mlsanity.com/paper/2503.00094/) (2025)
- [AutoODD: Agentic Audits via Bayesian Red Teaming in Black-Box Models](https://mlsanity.com/paper/2509.08638/) (2025)
- [Lipschitz Optimization for Formal Verification of Homographies](https://mlsanity.com/paper/2605.23203/) (2026)

## Abstract (arXiv)

Estimating the probability of failure is an important step in the certification of safety-critical systems. Efficient estimation methods are often needed due to the challenges posed by high-dimensional input spaces, risky test scenarios, and computationally expensive simulators. This work frames the problem of black-box safety validation as a Bayesian optimization problem and introduces a method that iteratively fits a probabilistic surrogate model to efficiently predict failures. The algorithm is designed to search for failures, compute the most-likely failure, and estimate the failure probability over an operating domain using importance sampling. We introduce three acquisition functions that aim to reduce uncertainty by covering the design space, optimize the analytically derived failure boundaries, and sample the predicted failure regions. Results show this Bayesian safety validation approach provides a more accurate estimate of failure probability with orders of magnitude fewer samples and performs well across various safety validation metrics. We demonstrate this approach on three test problems, a stochastic decision making system, and a neural network-based runway detection system. This work is open sourced (https://github.com/sisl/BayesianSafetyValidation.jl) and currently being used to supplement the FAA certification process of the machine learning components for an autonomous cargo aircraft.