# Sampling-based inference for large linear models, with application to linearised Laplace

arXiv: https://arxiv.org/abs/2210.04994 · HTML: https://mlsanity.com/paper/2210.04994/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 77 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 30 of 41)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019)
- [Efficiently Sampling Functions from Gaussian Process Posteriors](https://mlsanity.com/paper/2002.09309/) (2020)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Approximate Inference Turns Deep Networks into Gaussian Processes](https://mlsanity.com/paper/1906.01930/) (2019)
- [Bayesian Deep Learning via Subnetwork Inference](https://mlsanity.com/paper/2010.14689/) (2020)
- [Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression](https://mlsanity.com/paper/1602.05419/) (2016)
- [Fast Finite Width Neural Tangent Kernel](https://mlsanity.com/paper/2206.08720/) (2022)
- [Depth Uncertainty in Neural Networks](https://mlsanity.com/paper/2006.08437/) (2020)
- [Computed Tomography Reconstruction Using Deep Image Prior and Learned Reconstruction Methods](https://mlsanity.com/paper/2003.04989/) (2020)
- [Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited](https://mlsanity.com/paper/2003.02139/) (2020)
- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning](https://mlsanity.com/paper/1810.06530/) (2018)

## Built on by (most cited first, 18 of 18)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent](https://mlsanity.com/paper/2306.11589/) (2023)
- [Variational Linearized Laplace Approximation for Bayesian Deep Learning](https://mlsanity.com/paper/2302.12565/) (2023)
- [Stochastic Gradient Descent for Gaussian Processes Done Right](https://mlsanity.com/paper/2310.20581/) (2023)
- [Reparameterization invariance in approximate Bayesian inference](https://mlsanity.com/paper/2406.03334/) (2024)
- [Exploration via linearly perturbed loss minimisation](https://mlsanity.com/paper/2311.07565/) (2023)
- [FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning](https://mlsanity.com/paper/2407.13711/) (2024)
- [Uncertainty Quantification with the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2502.02870/) (2025)
- [Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization](https://mlsanity.com/paper/2304.08309/) (2023)
- [Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels](https://mlsanity.com/paper/2306.03968/) (2023)
- [Partially Stochastic Infinitely Deep Bayesian Neural Networks](https://mlsanity.com/paper/2402.03495/) (2024)
- [Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes](https://mlsanity.com/paper/2405.18457/) (2024)
- [Diffusion Active Learning: Towards Data-Driven Experimental Design in Computed Tomography](https://mlsanity.com/paper/2504.03491/) (2025)
- [Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projections](https://mlsanity.com/paper/2410.16901/) (2024)
- [Online Laplace Model Selection Revisited](https://mlsanity.com/paper/2307.06093/) (2023)
- [Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning](https://mlsanity.com/paper/2412.04177/) (2024)
- [Is the Last Layer Sufficient for Uncertainty Quantification?](https://mlsanity.com/paper/2605.30741/) (2026)
- [Warm Start Marginal Likelihood Optimisation for Iterative Gaussian Processes](https://mlsanity.com/paper/2405.18328/) (2024)

## Abstract (arXiv)

Large-scale linear models are ubiquitous throughout machine learning, with contemporary application as surrogate models for neural network uncertainty quantification; that is, the linearised Laplace method. Alas, the computational cost associated with Bayesian linear models constrains this method's application to small networks, small output spaces and small datasets. We address this limitation by introducing a scalable sample-based Bayesian inference method for conjugate Gaussian multi-output linear models, together with a matching method for hyperparameter (regularisation) selection. Furthermore, we use a classic feature normalisation method (the g-prior) to resolve a previously highlighted pathology of the linearised Laplace method. Together, these contributions allow us to perform linearised neural network inference with ResNet-18 on CIFAR100 (11M parameters, 100 outputs x 50k datapoints), with ResNet-50 on Imagenet (50M parameters, 1000 outputs x 1.2M datapoints) and with a U-Net on a high-resolution tomographic reconstruction task (2M parameters, 251k output~dimensions).