# Validation of Composite Systems by Discrepancy Propagation

arXiv: https://arxiv.org/abs/2210.12061 · HTML: https://mlsanity.com/paper/2210.12061/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Measuring Sample Quality with Kernels](https://mlsanity.com/paper/1703.01717/) (2017)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Bayesian Optimization of Composite Functions](https://mlsanity.com/paper/1906.01537/) (2019)
- [Bayesian Optimization of Function Networks](https://mlsanity.com/paper/2112.15311/) (2021)
- [Noisy-Input Entropy Search for Efficient Robust Bayesian Optimization](https://mlsanity.com/paper/2002.02820/) (2020)
- [Machine Learning for Reliability Engineering and Safety Applications: Review of Current Status and Future Opportunities](https://mlsanity.com/paper/2008.08221/) (2020)
- [BayesIMP: Uncertainty Quantification for Causal Data Fusion](https://mlsanity.com/paper/2106.03477/) (2021)
- [General Heuristics for Nonconvex Quadratically Constrained Quadratic Programming](https://mlsanity.com/paper/1703.07870/) (2017)
- [Bayesian Optimization for Cascade-type Multi-stage Processes](https://mlsanity.com/paper/2111.08330/) (2021)
- [Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction](https://mlsanity.com/paper/2008.06020/) (2020)
- [A Projection-free Algorithm for Constrained Stochastic Multi-level Composition Optimization](https://mlsanity.com/paper/2202.04296/) (2022)
- [Efficient sequential experimental design for surrogate modeling of nested codes](https://mlsanity.com/paper/1712.01620/) (2017)

## Built on by (most cited first, 2 of 2)

- [Attacking Motion Planners Using Adversarial Perception Errors](https://mlsanity.com/paper/2311.12722/) (2023)
- [Quantifying Local Model Validity using Active Learning](https://mlsanity.com/paper/2406.07474/) (2024)

## Abstract (arXiv)

Assessing the validity of a real-world system with respect to given quality criteria is a common yet costly task in industrial applications due to the vast number of required real-world tests. Validating such systems by means of simulation offers a promising and less expensive alternative, but requires an assessment of the simulation accuracy and therefore end-to-end measurements. Additionally, covariate shifts between simulations and actual usage can cause difficulties for estimating the reliability of such systems. In this work, we present a validation method that propagates bounds on distributional discrepancy measures through a composite system, thereby allowing us to derive an upper bound on the failure probability of the real system from potentially inaccurate simulations. Each propagation step entails an optimization problem, where -- for measures such as maximum mean discrepancy (MMD) -- we develop tight convex relaxations based on semidefinite programs. We demonstrate that our propagation method yields valid and useful bounds for composite systems exhibiting a variety of realistic effects. In particular, we show that the proposed method can successfully account for data shifts within the experimental design as well as model inaccuracies within the simulation.