mlsanityData to 7 Oct 2026

Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning

arXiv 2410.03390 ↗First posted 4 Oct 2024cs.CV cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox that allows the user to integrate UQ into their modelling workflow, without significant overhead. We introduce textttLightning UQ Box: a unified interface for applying and evaluating various approaches to UQ. In this paper, we provide a theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in our toolbox. We focus on two challenging vision tasks: (i) estimating tropical cyclone wind speeds from infrared satellite imagery and (ii) estimating the power output of solar panels from RGB images of the sky. By highlighting the differences between methods our results demonstrate the need for a broad and approachable experimental framework for UQ, that can be used for benchmarking UQ methods. The toolbox, example implementations, and further information are available at: https://github.com/lightning-uq-box/lightning-uq-box

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 40 of its 72 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

40 of 72 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  7. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  8. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  9. 2017Selective Classification for Deep Neural Networkscited by 799
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  11. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  12. 2019Conformalized Quantile Regressioncited by 620
28 more
  1. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  2. 2015Deep Kernel Learningcited by 440
  3. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  4. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  5. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  6. 2019Deep Evidential Regressioncited by 354
  7. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  8. 2020WeatherBench: A benchmark dataset for data-driven weather forecastingcited by 272
  9. 2022CARD: Classification and Regression Diffusion Modelscited by 102
  10. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  11. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  12. 2024Variational Bayesian Last Layerscited by 67
  13. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  14. 2020Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentationcited by 61
  15. 2021Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantificationcited by 57
  16. 2017Deep Learning in Robotics: A Review of Recent Researchcited by 42
  17. 2021GradDiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularizationcited by 34
  18. 2021Valid prediction intervals for regression problemscited by 28
  19. 2023Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' Viewcited by 27
  20. 2021Benchmarking of Deep Learning Irradiance Forecasting Models from Sky Images -- an in-depth Analysiscited by 21
  21. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  22. 2023Digital Typhoon: Long-term Satellite Image Dataset for the Spatio-Temporal Modeling of Tropical Cyclonescited by 19
  23. 2021ECLIPSE : Envisioning CLoud Induced Perturbations in Solar Energycited by 14
  24. 2023How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?cited by 14
  25. 2023Fortuna: A Library for Uncertainty Quantification in Deep Learningcited by 14
  26. 2022SKIPP'D: a SKy Images and Photovoltaic Power Generation Dataset for Short-term Solar Forecastingcited by 11
  27. 2021A framework for benchmarking uncertainty in deep regressioncited by 4
  28. 2024Uncertainty Aware Tropical Cyclone Wind Speed Estimation from Satellite Datacited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Learning Credal Ensembles via Distributionally Robust Optimizationcited by 6
  2. 2025Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantificationcited by 4
  3. 2026Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantificationcited by 0