# Variational Bayesian Last Layers

arXiv: https://arxiv.org/abs/2404.11599 · HTML: https://mlsanity.com/paper/2404.11599/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 86 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 30 of 52)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://mlsanity.com/paper/2106.09022/) (2021)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It](https://mlsanity.com/paper/1412.3730/) (2014)
- [Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/1611.00336/) (2016)
- [Bayesian Deep Learning via Subnetwork Inference](https://mlsanity.com/paper/2010.14689/) (2020)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- ['In-Between' Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1906.11537/) (2019)

## Built on by (most cited first, 30 of 69)

- [Offline Model-Based Optimization: Comprehensive Review](https://mlsanity.com/paper/2503.17286/) (2025)
- [Scalable Bayesian Learning with posteriors](https://mlsanity.com/paper/2406.00104/) (2024)
- [Training-Free Bayesianization for Low-Rank Adapters of Large Language Models](https://mlsanity.com/paper/2412.05723/) (2024)
- [Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles](https://mlsanity.com/paper/2412.15758/) (2024)
- [Uncertainty-Aware Reward Modeling for Stable RLHF](https://mlsanity.com/paper/2606.19818/) (2026)
- [Stochasticity in Motion: An Information-Theoretic Approach to Trajectory Prediction](https://mlsanity.com/paper/2410.01628/) (2024)
- [Closed-Form Last Layer Optimization](https://mlsanity.com/paper/2510.04606/) (2025)
- [Ranking over Regression for Bayesian Optimization and Molecule Selection](https://mlsanity.com/paper/2410.09290/) (2024)
- [Martingale Posterior Neural Networks for Fast Sequential Decision Making](https://mlsanity.com/paper/2506.11898/) (2025)
- [Variational Digital Twins](https://mlsanity.com/paper/2507.01047/) (2025)
- [Position: Biology is the Challenge Physics-Informed ML Needs to Evolve](https://mlsanity.com/paper/2510.25368/) (2025)
- [Richer Bayesian Last Layers with Subsampled NTK Features](https://mlsanity.com/paper/2602.01279/) (2026)
- [Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learning](https://mlsanity.com/paper/2410.03390/) (2024)
- [Uncertainty propagation in feed-forward neural network models](https://mlsanity.com/paper/2503.21059/) (2025)
- [Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models](https://mlsanity.com/paper/2407.14185/) (2024)
- [From Deep Additive Kernel Learning to Last-Layer Bayesian Neural Networks via Induced Prior Approximation](https://mlsanity.com/paper/2502.10540/) (2025)
- [Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer](https://mlsanity.com/paper/2505.22199/) (2025)
- [Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation](https://mlsanity.com/paper/2507.18366/) (2025)
- [Uncertainty-Aware Tabular Prediction: Evaluating VBLL-Enhanced TabPFN in Safety-Critical Medical Data](https://mlsanity.com/paper/2509.10048/) (2025)
- [Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimization](https://mlsanity.com/paper/2510.01471/) (2025)
- [CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regression](https://mlsanity.com/paper/2512.01384/) (2025)
- [Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimation](https://mlsanity.com/paper/2602.08142/) (2026)
- [Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapters](https://mlsanity.com/paper/2604.03388/) (2026)
- [Exploring Diffusion Models' Corruption Stage in Few-Shot Fine-tuning and Mitigating with Bayesian Neural Networks](https://mlsanity.com/paper/2405.19931/) (2024)
- [Flat Posterior Does Matter For Bayesian Model Averaging](https://mlsanity.com/paper/2406.15664/) (2024)
- [Last Layer Empirical Bayes](https://mlsanity.com/paper/2505.15888/) (2025)
- [Optimizing Data Augmentation through Bayesian Model Selection](https://mlsanity.com/paper/2505.21813/) (2025)
- [Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs](https://mlsanity.com/paper/2505.21857/) (2025)
- [Last Layer Hamiltonian Monte Carlo](https://mlsanity.com/paper/2507.08905/) (2025)
- [Similarity-Distance-Magnitude Activations](https://mlsanity.com/paper/2509.12760/) (2025)

## Abstract (arXiv)

We introduce a deterministic variational formulation for training Bayesian last layer neural networks. This yields a sampling-free, single-pass model and loss that effectively improves uncertainty estimation. Our variational Bayesian last layer (VBLL) can be trained and evaluated with only quadratic complexity in last layer width, and is thus (nearly) computationally free to add to standard architectures. We experimentally investigate VBLLs, and show that they improve predictive accuracy, calibration, and out of distribution detection over baselines across both regression and classification. Finally, we investigate combining VBLL layers with variational Bayesian feature learning, yielding a lower variance collapsed variational inference method for Bayesian neural networks.