mlsanityData to 7 Oct 2026

Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

arXiv 2109.10254 ↗First posted 21 Sep 2021cs.LG stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty. While the common goal in uncertainty quantification (UQ) in machine learning is to approximate the true distribution of the target data, many works in UQ tend to be disjoint in the evaluation metrics utilized, and disparate implementations for each metric lead to numerical results that are not directly comparable across different works. To address this, we introduce Uncertainty Toolbox, an open-source python library that helps to assess, visualize, and improve UQ. Uncertainty Toolbox additionally provides pedagogical resources, such as a glossary of key terms and an organized collection of key paper references. We hope that this toolbox is useful for accelerating and uniting research efforts in uncertainty in machine learning.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 21 Sep 2021.

Builds on

15 of 30 references.

References · 15

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  4. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 582
  5. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  6. 2018Uncertainty in Neural Networks: Approximately Bayesian Ensemblingcited by 94
  7. 2019Distribution Calibration for Regressioncited by 88
  8. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  9. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 77
  10. 2019Reliable training and estimation of variance networkscited by 63
  11. 2020Individual Calibration with Randomized Forecastingcited by 50
  12. 2020Calibrated Reliable Regression using Maximum Mean Discrepancycited by 36
3 more
  1. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  2. 2017Fast calibrated additive quantile regressioncited by 26
  3. 2017Calibration for the (Computationally-Identifiable) Massescited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 18; 2024: 14; 2025: 6; 2026: 9.

57 papers.

Sort:

By area:

  1. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  2. 2022NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operatorscited by 33
  3. 2023Collaborative Multi-Object Tracking with Conformal Uncertainty Propagationcited by 27
  4. 2023Bayesian Optimization of Catalysis With In-Context Learningcited by 20
  5. 2022Modular Conformal Calibrationcited by 15
  6. 2023Fortuna: A Library for Uncertainty Quantification in Deep Learningcited by 14
  7. 2023Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivitycited by 13
  8. 2022Generalizing Bayesian Optimization with Decision-theoretic Entropiescited by 11
  9. 2022Scalable Bayesian Uncertainty Quantification for Neural Network Potentials: Promise and Pitfallscited by 11
  10. 2022Uncertainty Quantification for Deep Unrolling-Based Computational Imagingcited by 10
  11. 2022Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Treescited by 9
  12. 2023A Benchmark on Uncertainty Quantification for Deep Learning Prognosticscited by 9
38 more
  1. 2024Full Shot Predictions for the DIII-D Tokamak via Deep Recurrent Networkscited by 9
  2. 2023Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantificationcited by 8
  3. 2024Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolutioncited by 8
  4. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  5. 2023On the role of Model Uncertainties in Bayesian Optimizationcited by 7
  6. 2025Light-Weight Diffusion Multiplier and Uncertainty Quantification for Fourier Neural Operatorscited by 6
  7. 2023Algorithmic Censoring in Dynamic Learning Systemscited by 5
  8. 2022A Survey on Uncertainty Toolkits for Deep Learningcited by 4
  9. 2023Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?cited by 4
  10. 2025Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantificationcited by 4
  11. 2022Tailored Uncertainty Estimation for Deep Learning Systemscited by 3
  12. 2023Multi-module based CVAE to predict HVCM faults in the SNS acceleratorcited by 3
  13. 2024Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learningcited by 3
  14. 2025Improved Uncertainty Quantification in Physics-Informed Neural Networks Using Error Bounds and Solution Bundlescited by 3
  15. 2023Surrogate uncertainty estimation for your time series forecasting black-box: learn when to trustcited by 2
  16. 2023Parity Calibrationcited by 2
  17. 2024Uncertainty Aware Tropical Cyclone Wind Speed Estimation from Satellite Datacited by 2
  18. 2024A Hybrid Probabilistic Battery Health Management Approach for Robust Inspection Drone Operationscited by 2
  19. 2024Towards certifiable AI in aviation: landscape, challenges, and opportunitiescited by 2
  20. 2024DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methodscited by 2
  21. 2023Distance Preserving Machine Learning for Uncertainty Aware Accelerator Capacitance Predictionscited by 1
  22. 2023Uncertainty Quantification for Image-based Traffic Prediction across Citiescited by 1
  23. 2023Challenges in data-based geospatial modeling for environmental research and practicecited by 1
  24. 2023Leveraging Uncertainty Estimates To Improve Classifier Performancecited by 1
  25. 2024Towards a Probabilistic Fusion Approach for Robust Battery Prognosticscited by 1
  26. 2024A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessmentcited by 1
  27. 2024Data-Driven Gradient Optimization for Field Emission Management in a Superconducting Radio-Frequency Linaccited by 1
  28. 2025Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Modelscited by 1
  29. 2021Diffusion Tensor Estimation with Uncertainty Calibrationcited by 0
  30. 2023Ensemble models outperform single model uncertainties and predictions for operator-learning of hypersonic flowscited by 0
  31. 2023Meta-learning to Calibrate Gaussian Processes with Deep Kernels for Regression Uncertainty Estimationcited by 0
  32. 2024Robust Calibration For Improved Weather Prediction Under Distributional Shiftcited by 0
  33. 2024Uncertainty Quantification in Alzheimer's Disease Progression Modelingcited by 0
  34. 2024Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shiftcited by 0
  35. 2025Uncertainty Guided Online Ensemble for Non-stationary Data Streams in Fusion Sciencecited by 0
  36. 2025Designing an Optimal Sensor Network via Minimizing Information Losscited by 0
  37. 2026A Critical Examination of Active Learning Workflows in Materials Sciencecited by 0
  38. 2026Self-Supervised Uncertainty Estimation For Super-Resolution of Satellite Imagescited by 0

50 most cited of 57 All 57 (JSON)