# Quantifying Local Model Validity using Active Learning

arXiv: https://arxiv.org/abs/2406.07474 · HTML: https://mlsanity.com/paper/2406.07474/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 75 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Bayesian Optimization in a Billion Dimensions via Random Embeddings](https://mlsanity.com/paper/1301.1942/) (2013)
- [Uniform Error Bounds for Gaussian Process Regression with Application to Safe Control](https://mlsanity.com/paper/1906.01376/) (2019)
- [Practical heteroskedastic Gaussian process modeling for large simulation experiments](https://mlsanity.com/paper/1611.05902/) (2016)
- [A Benchmark and Comparison of Active Learning for Logistic Regression](https://mlsanity.com/paper/1611.08618/) (2016)
- [Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information](https://mlsanity.com/paper/2104.09460/) (2021)
- [On the folded normal distribution](https://mlsanity.com/paper/1402.3559/) (2014)
- [Replication or exploration? Sequential design for stochastic simulation experiments](https://mlsanity.com/paper/1710.03206/) (2017)
- [Locally Valid and Discriminative Prediction Intervals for Deep Learning Models](https://mlsanity.com/paper/2106.00225/) (2021)
- [Bayesian Calibration of Imperfect Computer Models using Physics-Informed Priors](https://mlsanity.com/paper/2201.06463/) (2022)
- [Uniform Error and Posterior Variance Bounds for Gaussian Process Regression with Application to Safe Control](https://mlsanity.com/paper/2101.05328/) (2021)
- [Local Latin Hypercube Refinement for Multi-objective Design Uncertainty Optimization](https://mlsanity.com/paper/2108.08890/) (2021)
- [Validation of Composite Systems by Discrepancy Propagation](https://mlsanity.com/paper/2210.12061/) (2022)
- [Gradient and Uncertainty Enhanced Sequential Sampling for Global Fit](https://mlsanity.com/paper/2310.00110/) (2023)

## Abstract (arXiv)

Real-world applications of machine learning models are often subject to legal or policy-based regulations. Some of these regulations require ensuring the validity of the model, i.e., the approximation error being smaller than a threshold. A global metric is generally too insensitive to determine the validity of a specific prediction, whereas evaluating local validity is costly since it requires gathering additional data.We propose learning the model error to acquire a local validity estimate while reducing the amount of required data through active learning. Using model validation benchmarks, we provide empirical evidence that the proposed method can lead to an error model with sufficient discriminative properties using a relatively small amount of data. Furthermore, an increased sensitivity to local changes of the validity bounds compared to alternative approaches is demonstrated.