# Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning

arXiv: https://arxiv.org/abs/2410.03390 · HTML: https://mlsanity.com/paper/2410.03390/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 72 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [CARD: Classification and Regression Diffusion Models](https://mlsanity.com/paper/2206.07275/) (2022)
- [On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty](https://mlsanity.com/paper/2102.11409/) (2021)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation](https://mlsanity.com/paper/2002.09103/) (2020)
- [Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification](https://mlsanity.com/paper/2109.10254/) (2021)
- [Deep Learning in Robotics: A Review of Recent Research](https://mlsanity.com/paper/1707.07217/) (2017)
- [GradDiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularization](https://mlsanity.com/paper/2107.02425/) (2021)
- [Valid prediction intervals for regression problems](https://mlsanity.com/paper/2107.00363/) (2021)

## Built on by (most cited first, 3 of 3)

- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification](https://mlsanity.com/paper/2511.10282/) (2025)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)

## Abstract (arXiv)

Uncertainty quantification (UQ) is an essential tool for applying deep neural networks (DNNs) to real world tasks, as it attaches a degree of confidence to DNN outputs. However, despite its benefits, UQ is often left out of the standard DNN workflow due to the additional technical knowledge required to apply and evaluate existing UQ procedures. Hence there is a need for a comprehensive toolbox that allows the user to integrate UQ into their modelling workflow, without significant overhead. We introduce textttLightning UQ Box: a unified interface for applying and evaluating various approaches to UQ. In this paper, we provide a theoretical and quantitative comparison of the wide range of state-of-the-art UQ methods implemented in our toolbox. We focus on two challenging vision tasks: (i) estimating tropical cyclone wind speeds from infrared satellite imagery and (ii) estimating the power output of solar panels from RGB images of the sky. By highlighting the differences between methods our results demonstrate the need for a broad and approachable experimental framework for UQ, that can be used for benchmarking UQ methods. The toolbox, example implementations, and further information are available at: https://github.com/lightning-uq-box/lightning-uq-box