# A General Model Validation and Testing Tool

arXiv: https://arxiv.org/abs/1908.11251 · HTML: https://mlsanity.com/paper/1908.11251/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2019.

## Built on by (most cited first, 6 of 6)

- [Teaching Humans When To Defer to a Classifier via Exemplars](https://mlsanity.com/paper/2111.11297/) (2021)
- [GSR: A Generalized Symbolic Regression Approach](https://mlsanity.com/paper/2205.15569/) (2022)
- [Reliable Neural Networks for Regression Uncertainty Estimation](https://mlsanity.com/paper/2109.08213/) (2021)
- [MESSY Estimation: Maximum-Entropy based Stochastic and Symbolic densitY Estimation](https://mlsanity.com/paper/2306.04120/) (2023)
- [ISR: Invertible Symbolic Regression](https://mlsanity.com/paper/2405.06848/) (2024)
- [Inferential Moments of Uncertain Multivariable Systems](https://mlsanity.com/paper/2305.01841/) (2023)

## Abstract (arXiv)

We construct and propose the "Bayesian Validation Metric" (BVM) as a general model validation and testing tool. We find the BVM to be capable of representing all of the standard validation metrics (square error, reliability, probability of agreement, frequentist, area, probability density comparison, statistical hypothesis testing, and Bayesian model testing) as special cases and find that it can be used to improve, generalize, or further quantify their uncertainties. Thus, the BVM allows us to assess the similarities and differences between existing validation metrics in a new light. The BVM has the capacity to allow users to invent and select models according to novel validation requirements. We formulate and test a few novel compound validation metrics that improve upon other validation metrics in the literature. Further, we construct the BVM Ratio for the purpose of quantifying model selection under user defined definitions of agreement in the presence or absence of uncertainty. This construction generalizes the Bayesian model testing framework.