# On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation

arXiv: https://arxiv.org/abs/2001.08049 · HTML: https://mlsanity.com/paper/2001.08049/

## Key facts

- Cited by 23 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 22 Jan 2020.

## Built on by (most cited first, 23 of 23)

- [Deep Ensembles from a Bayesian Perspective](https://mlsanity.com/paper/2105.13283/) (2021)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [Robust Semantic Segmentation with Superpixel-Mix](https://mlsanity.com/paper/2108.00968/) (2021)
- [On double-descent in uncertainty quantification in overparametrized models](https://mlsanity.com/paper/2210.12760/) (2022)
- [Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models](https://mlsanity.com/paper/2312.15297/) (2023)
- [Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle](https://mlsanity.com/paper/2105.14559/) (2021)
- [A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2310.08287/) (2023)
- [Bayesian Neural Networks: Essentials](https://mlsanity.com/paper/2106.13594/) (2021)
- [Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Review](https://mlsanity.com/paper/2312.09454/) (2023)
- [Guided Deep Kernel Learning](https://mlsanity.com/paper/2302.09574/) (2023)
- [Bayesian Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2408.01694/) (2024)
- [Density Uncertainty Layers for Reliable Uncertainty Estimation](https://mlsanity.com/paper/2306.12497/) (2023)
- [Neural density estimation and uncertainty quantification for laser induced breakdown spectroscopy spectra](https://mlsanity.com/paper/2108.08709/) (2021)
- [Bayesian neural networks for the probabilistic forecasting of wind direction and speed using ocean data](https://mlsanity.com/paper/2206.08953/) (2022)
- [Bayesian Learning for Disparity Map Refinement for Semi-Dense Active Stereo Vision](https://mlsanity.com/paper/2209.05082/) (2022)
- [Is the Last Layer Sufficient for Uncertainty Quantification?](https://mlsanity.com/paper/2605.30741/) (2026)
- [Model Architecture Adaption for Bayesian Neural Networks](https://mlsanity.com/paper/2202.04392/) (2022)
- [Variational Inference on the Final-Layer Output of Neural Networks](https://mlsanity.com/paper/2302.02420/) (2023)
- [Last layer state space model for representation learning and uncertainty quantification](https://mlsanity.com/paper/2307.01566/) (2023)
- [B-PL-PINN: Stabilizing PINN Training with Bayesian Pseudo Labeling](https://mlsanity.com/paper/2507.01714/) (2025)
- [Partial Trace-Class Bayesian Neural Networks](https://mlsanity.com/paper/2511.01628/) (2025)
- [Bayesian Complete-Pooling in Cross-Subject Classification for Motor Imagery Electroencephalogram](https://mlsanity.com/paper/2607.22980/) (2026)
- [Uncertainty quantification for trustworthy deep learning: Methods and measures](https://mlsanity.com/paper/2607.28248/) (2026)

## Abstract (arXiv)

Uncertainty quantification for deep learning is a challenging open problem. Bayesian statistics offer a mathematically grounded framework to reason about uncertainties; however, approximate posteriors for modern neural networks still require prohibitive computational costs. We propose a family of algorithms which split the classification task into two stages: representation learning and uncertainty estimation. We compare four specific instances, where uncertainty estimation is performed via either an ensemble of Stochastic Gradient Descent or Stochastic Gradient Langevin Dynamics snapshots, an ensemble of bootstrapped logistic regressions, or via a number of Monte Carlo Dropout passes. We evaluate their performance in terms of selective classification (risk-coverage), and their ability to detect out-of-distribution samples. Our experiments suggest there is limited value in adding multiple uncertainty layers to deep classifiers, and we observe that these simple methods strongly outperform a vanilla point-estimate SGD in some complex benchmarks like ImageNet.