# Exploring the Limits of Epistemic Uncertainty Quantification in Low-Shot Settings

arXiv: https://arxiv.org/abs/2111.09808 · HTML: https://mlsanity.com/paper/2111.09808/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches](https://mlsanity.com/paper/1803.04386/) (2018)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks](https://mlsanity.com/paper/1906.04569/) (2019)
- [The Case for Bayesian Deep Learning](https://mlsanity.com/paper/2001.10995/) (2020)
- [Classification Uncertainty of Deep Neural Networks Based on Gradient Information](https://mlsanity.com/paper/1805.08440/) (2018)

## Built on by (most cited first, 4 of 4)

- [Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Review](https://mlsanity.com/paper/2312.09454/) (2023)
- [Measuring Orthogonality as the Blind-Spot of Uncertainty Disentanglement](https://mlsanity.com/paper/2408.12175/) (2024)
- [Unreliable Uncertainty Estimates with Monte Carlo Dropout](https://mlsanity.com/paper/2512.14851/) (2025)
- [Modeling of AUV Dynamics with Limited Resources: Efficient Online Learning Using Uncertainty](https://mlsanity.com/paper/2504.04583/) (2025)

## Abstract (arXiv)

Uncertainty quantification in neural network promises to increase safety of AI systems, but it is not clear how performance might vary with the training set size. In this paper we evaluate seven uncertainty methods on Fashion MNIST and CIFAR10, as we sub-sample and produce varied training set sizes. We find that calibration error and out of distribution detection performance strongly depend on the training set size, with most methods being miscalibrated on the test set with small training sets. Gradient-based methods seem to poorly estimate epistemic uncertainty and are the most affected by training set size. We expect our results can guide future research into uncertainty quantification and help practitioners select methods based on their particular available data.