# Variational Filtering for Bayesian Last Layers with an Independent Binary Probit Surrogate

arXiv: https://arxiv.org/abs/2610.16578 · HTML: https://mlsanity.com/paper/2610.16578/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 26 of 26)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Equinox: neural networks in JAX via callable PyTrees and filtered transformations](https://mlsanity.com/paper/2111.00254/) (2021)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Streaming Variational Bayes](https://mlsanity.com/paper/1307.6769/) (2013)
- [Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration](https://mlsanity.com/paper/2512.22382/) (2025)
- [Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentation](https://mlsanity.com/paper/1506.05843/) (2015)
- [Kalman Filter for Online Classification of Non-Stationary Data](https://mlsanity.com/paper/2306.08448/) (2023)
- [Improved uncertainty quantification for neural networks with Bayesian last layer](https://mlsanity.com/paper/2302.10975/) (2023)
- [An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence](https://mlsanity.com/paper/2010.02709/) (2020)
- [SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks](https://mlsanity.com/paper/2606.23357/) (2026)
- [Easy Variational Inference for Categorical Models via an Independent Binary Approximation](https://mlsanity.com/paper/2206.00093/) (2022)

## Abstract (arXiv)

Modern deep classifiers trained by gradient descent need many passes over the data to become accurate, and even then are not always well calibrated; post-hoc temperature scaling corrects calibration after training but cannot influence training-time accuracy or convergence. We propose a variational filtering algorithm for the last layer of deep classifiers, whose parameters are treated as the latent state of a linear-Gaussian state-space model. Each mini-batch is assimilated by an exact prediction step followed by a closed-form variational update, made conjugate by an independent binary probit (IBProbit) surrogate for the categorical likelihood. The filter's process noise stabilises the updates under feature drift and keeps the predictive distribution from collapsing to a point estimate, allowing the same procedure to be used for linear probing on a frozen backbone and for full-network fine-tuning. We evaluate on DINOv3 backbones against two baselines tuned under an identical protocol: AdamW with cross-entropy and post-hoc temperature scaling, and Variational Bayesian Last Layers (VBLL). As a linear probe on frozen features, IBProbit attains the best accuracy and negative log-likelihood on all five downstream tasks at the largest backbone, and on two of the five at the smallest. In out-of-distribution detection with CIFAR probes on the largest backbone, IBProbit reduces the false-positive rate at 95% true-positive rate by a factor of 1.4 to 3.2 relative to the better of the two baselines, on all three dataset pairs. When the whole network is fine-tuned on ImageNet-1k, IBProbit reaches the final performance of the cross-entropy baseline in half the wall-clock time.