# Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

arXiv: https://arxiv.org/abs/2109.10254 · HTML: https://mlsanity.com/paper/2109.10254/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [Calibrated Reliable Regression using Maximum Mean Discrepancy](https://mlsanity.com/paper/2006.10255/) (2020)
- [Calibrated Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/1906.08312/) (2019)
- [Fast calibrated additive quantile regression](https://mlsanity.com/paper/1707.03307/) (2017)
- [Calibration for the (Computationally-Identifiable) Masses](https://mlsanity.com/paper/1711.08513/) (2017)

## Built on by (most cited first, 30 of 57)

- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators](https://mlsanity.com/paper/2208.11866/) (2022)
- [Collaborative Multi-Object Tracking with Conformal Uncertainty Propagation](https://mlsanity.com/paper/2303.14346/) (2023)
- [Bayesian Optimization of Catalysis With In-Context Learning](https://mlsanity.com/paper/2304.05341/) (2023)
- [Modular Conformal Calibration](https://mlsanity.com/paper/2206.11468/) (2022)
- [Fortuna: A Library for Uncertainty Quantification in Deep Learning](https://mlsanity.com/paper/2302.04019/) (2023)
- [Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity](https://mlsanity.com/paper/2309.06240/) (2023)
- [Generalizing Bayesian Optimization with Decision-theoretic Entropies](https://mlsanity.com/paper/2210.01383/) (2022)
- [Scalable Bayesian Uncertainty Quantification for Neural Network Potentials: Promise and Pitfalls](https://mlsanity.com/paper/2212.07959/) (2022)
- [Uncertainty Quantification for Deep Unrolling-Based Computational Imaging](https://mlsanity.com/paper/2207.00698/) (2022)
- [Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees](https://mlsanity.com/paper/2205.11412/) (2022)
- [A Benchmark on Uncertainty Quantification for Deep Learning Prognostics](https://mlsanity.com/paper/2302.04730/) (2023)
- [Full Shot Predictions for the DIII-D Tokamak via Deep Recurrent Networks](https://mlsanity.com/paper/2404.12416/) (2024)
- [On the role of Model Uncertainties in Bayesian Optimization](https://mlsanity.com/paper/2301.05983/) (2023)
- [Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantification](https://mlsanity.com/paper/2302.02595/) (2023)
- [Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution](https://mlsanity.com/paper/2402.08383/) (2024)
- [Language Model Embeddings Can Be Sufficient for Bayesian Optimization](https://mlsanity.com/paper/2410.10190/) (2024)
- [Light-Weight Diffusion Multiplier and Uncertainty Quantification for Fourier Neural Operators](https://mlsanity.com/paper/2508.00643/) (2025)
- [Algorithmic Censoring in Dynamic Learning Systems](https://mlsanity.com/paper/2305.09035/) (2023)
- [A Survey on Uncertainty Toolkits for Deep Learning](https://mlsanity.com/paper/2205.01040/) (2022)
- [Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?](https://mlsanity.com/paper/2310.11978/) (2023)
- [Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification](https://mlsanity.com/paper/2511.10282/) (2025)
- [Tailored Uncertainty Estimation for Deep Learning Systems](https://mlsanity.com/paper/2204.13963/) (2022)
- [Multi-module based CVAE to predict HVCM faults in the SNS accelerator](https://mlsanity.com/paper/2304.10639/) (2023)
- [Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning](https://mlsanity.com/paper/2410.03390/) (2024)
- [Improved Uncertainty Quantification in Physics-Informed Neural Networks Using Error Bounds and Solution Bundles](https://mlsanity.com/paper/2505.06459/) (2025)
- [Surrogate uncertainty estimation for your time series forecasting black-box: learn when to trust](https://mlsanity.com/paper/2302.02834/) (2023)
- [Parity Calibration](https://mlsanity.com/paper/2305.18655/) (2023)
- [Uncertainty Aware Tropical Cyclone Wind Speed Estimation from Satellite Data](https://mlsanity.com/paper/2404.08325/) (2024)
- [A Hybrid Probabilistic Battery Health Management Approach for Robust Inspection Drone Operations](https://mlsanity.com/paper/2405.00055/) (2024)

## Abstract (arXiv)

With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty. While the common goal in uncertainty quantification (UQ) in machine learning is to approximate the true distribution of the target data, many works in UQ tend to be disjoint in the evaluation metrics utilized, and disparate implementations for each metric lead to numerical results that are not directly comparable across different works. To address this, we introduce Uncertainty Toolbox, an open-source python library that helps to assess, visualize, and improve UQ. Uncertainty Toolbox additionally provides pedagogical resources, such as a glossary of key terms and an organized collection of key paper references. We hope that this toolbox is useful for accelerating and uniting research efforts in uncertainty in machine learning.