# Unknown Examples & Machine Learning Model Generalization

arXiv: https://arxiv.org/abs/1808.08294 · HTML: https://mlsanity.com/paper/1808.08294/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2018.

## Built on by (most cited first, 5 of 5)

- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [On The Reliability Of Machine Learning Applications In Manufacturing Environments](https://mlsanity.com/paper/2112.06986/) (2021)
- [ESCAPE: Countering Systematic Errors from Machine's Blind Spots via Interactive Visual Analysis](https://mlsanity.com/paper/2303.09657/) (2023)
- [Probabilistic analysis of solar cell optical performance using Gaussian processes](https://mlsanity.com/paper/2107.07342/) (2021)
- [Application of Deep Learning for Predictive Maintenance of Oilfield Equipment](https://mlsanity.com/paper/2306.11040/) (2023)

## Abstract (arXiv)

Over the past decades, researchers and ML practitioners have come up with better and better ways to build, understand and improve the quality of ML models, but mostly under the key assumption that the training data is distributed identically to the testing data. In many real-world applications, however, some potential training examples are unknown to the modeler, due to sample selection bias or, more generally, covariate shift, i.e., a distribution shift between the training and deployment stage. The resulting discrepancy between training and testing distributions leads to poor generalization performance of the ML model and hence biased predictions. We provide novel algorithms that estimate the number and properties of these unknown training examples---unknown unknowns. This information can then be used to correct the training set, prior to seeing any test data. The key idea is to combine species-estimation techniques with data-driven methods for estimating the feature values for the unknown unknowns. Experiments on a variety of ML models and datasets indicate that taking the unknown examples into account can yield a more robust ML model that generalizes better.