# Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning

arXiv: https://arxiv.org/abs/2412.04177 · HTML: https://mlsanity.com/paper/2412.04177/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 59 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches](https://mlsanity.com/paper/1803.04386/) (2018)
- [Approximate Inference Turns Deep Networks into Gaussian Processes](https://mlsanity.com/paper/1906.01930/) (2019)
- [Bayesian Deep Learning via Subnetwork Inference](https://mlsanity.com/paper/2010.14689/) (2020)
- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Sparse Gaussian Processes with Spherical Harmonic Features](https://mlsanity.com/paper/2006.16649/) (2020)
- [Black-box α-divergence Minimization](https://mlsanity.com/paper/1511.03243/) (2015)
- [Dropout Inference in Bayesian Neural Networks with Alpha-divergences](https://mlsanity.com/paper/1703.02914/) (2017)
- [Accelerated Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2210.12642/) (2022)
- [Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent](https://mlsanity.com/paper/2306.11589/) (2023)
- [Variational Inference for Gaussian Process Models with Linear Complexity](https://mlsanity.com/paper/1711.10127/) (2017)
- [Sampling-based inference for large linear models, with application to linearised Laplace](https://mlsanity.com/paper/2210.04994/) (2022)
- [Variational Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2302.12565/) (2023)
- [BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning](https://mlsanity.com/paper/2010.01979/) (2020)
- [Deep Variational Implicit Processes](https://mlsanity.com/paper/2206.06720/) (2022)
- [Function-space Parameterization of Neural Networks for Sequential Learning](https://mlsanity.com/paper/2403.10929/) (2024)
- [The Gaussian Radon Transform and Machine Learning](https://mlsanity.com/paper/1310.4794/) (2013)

## Built on by (most cited first, 2 of 2)

- [Richer Bayesian Last Layers with Subsampled NTK Features](https://mlsanity.com/paper/2602.01279/) (2026)
- [Scalable Linearized Laplace Approximation via Surrogate Neural Kernel](https://mlsanity.com/paper/2601.21835/) (2026)

## Abstract (arXiv)

Recently, there has been an increasing interest in performing post-hoc uncertainty estimation about the predictions of pre-trained deep neural networks (DNNs). Given a pre-trained DNN via back-propagation, these methods enhance the original network by adding output confidence measures, such as error bars, without compromising its initial accuracy. In this context, we introduce a novel family of sparse variational Gaussian processes (GPs), where the posterior mean is fixed to any continuous function when using a universal kernel. Specifically, we fix the mean of this GP to the output of the pre-trained DNN, allowing our approach to effectively fit the GP's predictive variances to estimate the DNN prediction uncertainty. Our approach leverages variational inference (VI) for efficient stochastic optimization, with training costs that remain independent of the number of training points, scaling efficiently to large datasets such as ImageNet. The proposed method, called fixed-mean GP (FMGP), is architecture-agnostic, relying solely on the pre-trained model's outputs to adjust the predictive variances. Experimental results demonstrate that FMGP improves both uncertainty estimation and computational efficiency when compared to state-of-the-art methods for DNN post-hoc Bayesian inference.