# Uncertainty Quantification for Rule-Based Models

arXiv: https://arxiv.org/abs/2211.01915 · HTML: https://mlsanity.com/paper/2211.01915/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Optimal strategies for reject option classifiers](https://mlsanity.com/paper/2101.12523/) (2021)
- [A Swiss Army Infinitesimal Jackknife](https://mlsanity.com/paper/1806.00550/) (2018)
- [Beyond Perturbations: Learning Guarantees with Arbitrary Adversarial Test Examples](https://mlsanity.com/paper/2007.05145/) (2020)
- [Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes](https://mlsanity.com/paper/1805.05396/) (2018)
- [Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors](https://mlsanity.com/paper/1909.04079/) (2019)
- [Learning Prediction Intervals for Model Performance](https://mlsanity.com/paper/2012.08625/) (2020)
- [Uncertainty Characteristics Curves: A Systematic Assessment of Prediction Intervals](https://mlsanity.com/paper/2106.00858/) (2021)
- [The Utility of Abstaining in Binary Classification](https://mlsanity.com/paper/1512.08133/) (2015)
- [Towards optimally abstaining from prediction with OOD test examples](https://mlsanity.com/paper/2105.14119/) (2021)

## Abstract (arXiv)

Rule-based classification models described in the language of logic directly predict boolean values, rather than modeling a probability and translating it into a prediction as done in statistical models. The vast majority of existing uncertainty quantification approaches rely on models providing continuous output not available to rule-based models. In this work, we propose an uncertainty quantification framework in the form of a meta-model that takes any binary classifier with binary output as a black box and estimates the prediction accuracy of that base model at a given input along with a level of confidence on that estimation. The confidence is based on how well that input region is explored and is designed to work in any OOD scenario. We demonstrate the usefulness of this uncertainty model by building an abstaining classifier powered by it and observing its performance in various scenarios.