# Lightweight Probabilistic Deep Networks

arXiv: https://arxiv.org/abs/1805.11327 · HTML: https://mlsanity.com/paper/1805.11327/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2018.

## Built on by (most cited first, 30 of 77)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer](https://mlsanity.com/paper/2208.14201/) (2022)
- [Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation](https://mlsanity.com/paper/2109.09881/) (2021)
- [Learning Accurate Dense Correspondences and When to Trust Them](https://mlsanity.com/paper/2101.01710/) (2021)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [PDC-Net+: Enhanced Probabilistic Dense Correspondence Network](https://mlsanity.com/paper/2109.13912/) (2021)
- [Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression](https://mlsanity.com/paper/2103.13629/) (2021)
- [Shapley Explanation Networks](https://mlsanity.com/paper/2104.02297/) (2021)
- [Global canopy height regression and uncertainty estimation from GEDI LIDAR waveforms with deep ensembles](https://mlsanity.com/paper/2103.03975/) (2021)
- [Bayesian Deep Basis Fitting for Depth Completion with Uncertainty](https://mlsanity.com/paper/2103.15254/) (2021)
- [ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference](https://mlsanity.com/paper/2211.11435/) (2022)
- [DUDES: Deep Uncertainty Distillation using Ensembles for Semantic Segmentation](https://mlsanity.com/paper/2303.09843/) (2023)
- [Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributions](https://mlsanity.com/paper/2105.04471/) (2021)
- [SLURP: Side Learning Uncertainty for Regression Problems](https://mlsanity.com/paper/2110.11182/) (2021)
- [A General Framework for quantifying Aleatoric and Epistemic uncertainty in Graph Neural Networks](https://mlsanity.com/paper/2205.09968/) (2022)
- [An Analytic Solution to Covariance Propagation in Neural Networks](https://mlsanity.com/paper/2403.16163/) (2024)
- [Uncertainty Estimation for Molecules: Desiderata and Methods](https://mlsanity.com/paper/2306.14916/) (2023)
- [U-CE: Uncertainty-aware Cross-Entropy for Semantic Segmentation](https://mlsanity.com/paper/2307.09947/) (2023)
- [Keep the Faith: Faithful Explanations in Convolutional Neural Networks for Case-Based Reasoning](https://mlsanity.com/paper/2312.09783/) (2023)
- [Uncertainty Quantification via Stable Distribution Propagation](https://mlsanity.com/paper/2402.08324/) (2024)
- [Country-wide Retrieval of Forest Structure From Optical and SAR Satellite Imagery With Deep Ensembles](https://mlsanity.com/paper/2111.13154/) (2021)
- [Have We Ever Encountered This Before? Retrieving Out-of-Distribution Road Obstacles from Driving Scenes](https://mlsanity.com/paper/2309.04302/) (2023)
- [Driving Through Ghosts: Behavioral Cloning with False Positives](https://mlsanity.com/paper/2008.12969/) (2020)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Review](https://mlsanity.com/paper/2312.09454/) (2023)
- [MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints](https://mlsanity.com/paper/2109.06768/) (2021)
- [NudgeSeg: Zero-Shot Object Segmentation by Repeated Physical Interaction](https://mlsanity.com/paper/2109.13859/) (2021)
- [Data Models for Dataset Drift Controls in Machine Learning With Optical Images](https://mlsanity.com/paper/2211.02578/) (2022)
- [SOTIF Entropy: Online SOTIF Risk Quantification and Mitigation for Autonomous Driving](https://mlsanity.com/paper/2211.04009/) (2022)
- [Propagation and Attribution of Uncertainty in Medical Imaging Pipelines](https://mlsanity.com/paper/2309.16831/) (2023)

## Abstract (arXiv)

Even though probabilistic treatments of neural networks have a long history, they have not found widespread use in practice. Sampling approaches are often too slow already for simple networks. The size of the inputs and the depth of typical CNN architectures in computer vision only compound this problem. Uncertainty in neural networks has thus been largely ignored in practice, despite the fact that it may provide important information about the reliability of predictions and the inner workings of the network. In this paper, we introduce two lightweight approaches to making supervised learning with probabilistic deep networks practical: First, we suggest probabilistic output layers for classification and regression that require only minimal changes to existing networks. Second, we employ assumed density filtering and show that activation uncertainties can be propagated in a practical fashion through the entire network, again with minor changes. Both probabilistic networks retain the predictive power of the deterministic counterpart, but yield uncertainties that correlate well with the empirical error induced by their predictions. Moreover, the robustness to adversarial examples is significantly increased.