# Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI

arXiv: https://arxiv.org/abs/2106.01410 · HTML: https://mlsanity.com/paper/2106.01410/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Model Selection in Bayesian Neural Networks via Horseshoe Priors](https://mlsanity.com/paper/1705.10388/) (2017)
- [Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes](https://mlsanity.com/paper/1805.05396/) (2018)
- [Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors](https://mlsanity.com/paper/1909.04079/) (2019)
- [Approximate Cross-Validation for Structured Models](https://mlsanity.com/paper/2006.12669/) (2020)
- [Learning Prediction Intervals for Model Performance](https://mlsanity.com/paper/2012.08625/) (2020)
- [Uncertainty Prediction for Deep Sequential Regression Using Meta Models](https://mlsanity.com/paper/2007.01350/) (2020)
- [Uncertainty Characteristics Curves: A Systematic Assessment of Prediction Intervals](https://mlsanity.com/paper/2106.00858/) (2021)

## Built on by (most cited first, 22 of 22)

- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](https://mlsanity.com/paper/2110.10790/) (2021)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Investigating Explainability of Generative AI for Code through Scenario-based Design](https://mlsanity.com/paper/2202.04903/) (2022)
- [Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI](https://mlsanity.com/paper/2206.10847/) (2022)
- [Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making](https://mlsanity.com/paper/2403.16812/) (2024)
- [Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits](https://mlsanity.com/paper/2205.06922/) (2022)
- [As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making](https://mlsanity.com/paper/2501.12868/) (2025)
- [JAWS: Auditing Predictive Uncertainty Under Covariate Shift](https://mlsanity.com/paper/2207.10716/) (2022)
- [Fortuna: A Library for Uncertainty Quantification in Deep Learning](https://mlsanity.com/paper/2302.04019/) (2023)
- [TyXe: Pyro-based Bayesian neural nets for Pytorch](https://mlsanity.com/paper/2110.00276/) (2021)
- [Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacy](https://mlsanity.com/paper/2402.07632/) (2024)
- [What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization](https://mlsanity.com/paper/2207.05161/) (2022)
- [A Survey on Uncertainty Toolkits for Deep Learning](https://mlsanity.com/paper/2205.01040/) (2022)
- [Quantitative AI Risk Assessments: Opportunities and Challenges](https://mlsanity.com/paper/2209.06317/) (2022)
- [Tailored Uncertainty Estimation for Deep Learning Systems](https://mlsanity.com/paper/2204.13963/) (2022)
- [Explainable Global Error Weighted on Feature Importance: The xGEWFI metric to evaluate the error of data imputation and data augmentation](https://mlsanity.com/paper/2206.08980/) (2022)
- [Deep Surrogate of Modular Multi Pump using Active Learning](https://mlsanity.com/paper/2208.02840/) (2022)
- [Measuring the Confidence of Traffic Forecasting Models: Techniques, Experimental Comparison and Guidelines towards Their Actionability](https://mlsanity.com/paper/2210.16049/) (2022)
- [Function Composition in Trustworthy Machine Learning: Implementation Choices, Insights, and Questions](https://mlsanity.com/paper/2302.09190/) (2023)
- [A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment](https://mlsanity.com/paper/2408.01301/) (2024)
- [Neurosymbolic Artificial Intelligence for Robust Network Intrusion Detection: From Scratch to Transfer Learning](https://mlsanity.com/paper/2506.04454/) (2025)
- [Bridging Ethical Principles and Algorithmic Methods: An Alternative Approach for Assessing Trustworthiness in AI Systems](https://mlsanity.com/paper/2506.22774/) (2025)

## Abstract (arXiv)

In this paper, we describe an open source Python toolkit named Uncertainty Quantification 360 (UQ360) for the uncertainty quantification of AI models. The goal of this toolkit is twofold: first, to provide a broad range of capabilities to streamline as well as foster the common practices of quantifying, evaluating, improving, and communicating uncertainty in the AI application development lifecycle; second, to encourage further exploration of UQ's connections to other pillars of trustworthy AI such as fairness and transparency through the dissemination of latest research and education materials. Beyond the Python package (urlhttps://github.com/IBM/UQ360), we have developed an interactive experience (urlhttp://uq360.mybluemix.net) and guidance materials as educational tools to aid researchers and developers in producing and communicating high-quality uncertainties in an effective manner.