# Distributionally Robust Statistical Verification with Imprecise Neural Networks

arXiv: https://arxiv.org/abs/2308.14815 · HTML: https://mlsanity.com/paper/2308.14815/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 85 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 45)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems](https://mlsanity.com/paper/2004.05519/) (2020)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Robust Validation: Confident Predictions Even When Distributions Shift](https://mlsanity.com/paper/2008.04267/) (2020)
- [Distribution-free binary classification: prediction sets, confidence intervals and calibration](https://mlsanity.com/paper/2006.10564/) (2020)
- [A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems](https://mlsanity.com/paper/2005.02979/) (2020)
- [An Abstraction-Based Framework for Neural Network Verification](https://mlsanity.com/paper/1910.14574/) (2019)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Probabilistic Safety for Bayesian Neural Networks](https://mlsanity.com/paper/2004.10281/) (2020)
- [Sample-Efficient Safety Assurances using Conformal Prediction](https://mlsanity.com/paper/2109.14082/) (2021)
- [OVERT: An Algorithm for Safety Verification of Neural Network Control Policies for Nonlinear Systems](https://mlsanity.com/paper/2108.01220/) (2021)
- [Statistical Guarantees for the Robustness of Bayesian Neural Networks](https://mlsanity.com/paper/1903.01980/) (2019)
- [Conformal Prediction for STL Runtime Verification](https://mlsanity.com/paper/2211.01539/) (2022)
- [A System-Level View on Out-of-Distribution Data in Robotics](https://mlsanity.com/paper/2212.14020/) (2022)
- [DeepAbstract: Neural Network Abstraction for Accelerating Verification](https://mlsanity.com/paper/2006.13735/) (2020)
- [Multi-Agent Reachability Calibration with Conformal Prediction](https://mlsanity.com/paper/2304.00432/) (2023)
- [On Linear Optimization over Wasserstein Balls](https://mlsanity.com/paper/2004.07162/) (2020)
- [Learning Representations for Pixel-based Control: What Matters and Why?](https://mlsanity.com/paper/2111.07775/) (2021)
- [PAC Confidence Predictions for Deep Neural Network Classifiers](https://mlsanity.com/paper/2011.00716/) (2020)
- [Probabilistic Verification and Reachability Analysis of Neural Networks via Semidefinite Programming](https://mlsanity.com/paper/1910.04249/) (2019)
- [Interval Neural Networks: Uncertainty Scores](https://mlsanity.com/paper/2003.11566/) (2020)
- [Pointwise Feasibility of Gaussian Process-based Safety-Critical Control under Model Uncertainty](https://mlsanity.com/paper/2106.07108/) (2021)
- [A Novel Bayes' Theorem for Upper Probabilities](https://mlsanity.com/paper/2307.06831/) (2023)

## Built on by (most cited first, 12 of 12)

- [Conformalized Credal Regions for Classification with Ambiguous Ground Truth](https://mlsanity.com/paper/2411.04852/) (2024)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [Integral Imprecise Probability Metrics](https://mlsanity.com/paper/2505.16156/) (2025)
- [Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions](https://mlsanity.com/paper/2411.00465/) (2024)
- [A Unified Approach Towards Active Learning and Out-of-Distribution Detection](https://mlsanity.com/paper/2405.11337/) (2024)
- [Conformal Prediction Regions are Imprecise Highest Density Regions](https://mlsanity.com/paper/2502.06331/) (2025)
- [Reasoning with random sets: An agenda for the future](https://mlsanity.com/paper/2401.09435/) (2023)
- [Bridging Dimensions: Confident Reachability for High-Dimensional Controllers](https://mlsanity.com/paper/2311.04843/) (2023)
- [Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry](https://mlsanity.com/paper/2602.01667/) (2026)
- [Adaptive Conformal Guidance for Learning under Uncertainty](https://mlsanity.com/paper/2502.16736/) (2025)
- [When Do Credal Sets Stabilize? Fixed-Point Theorems for Credal Set Updates](https://mlsanity.com/paper/2510.04769/) (2025)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)

## Abstract (arXiv)

A particularly challenging problem in AI safety is providing guarantees on the behavior of high-dimensional autonomous systems. Verification approaches centered around reachability analysis fail to scale, and purely statistical approaches are constrained by the distributional assumptions about the sampling process. Instead, we pose a distributionally robust version of the statistical verification problem for black-box systems, where our performance guarantees hold over a large family of distributions. This paper proposes a novel approach based on uncertainty quantification using concepts from imprecise probabilities. A central piece of our approach is an ensemble technique called Imprecise Neural Networks, which provides the uncertainty quantification. Additionally, we solve the allied problem of exploring the input set using active learning. The active learning uses an exhaustive neural-network verification tool Sherlock to collect samples. An evaluation on multiple physical simulators in the openAI gym Mujoco environments with reinforcement-learned controllers demonstrates that our approach can provide useful and scalable guarantees for high-dimensional systems.