mlsanityData to 7 Oct 2026

Sampling-based inference for large linear models, with application to linearised Laplace

arXiv 2210.04994 ↗First posted 10 Oct 2022stat.ML cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Large-scale linear models are ubiquitous throughout machine learning, with contemporary application as surrogate models for neural network uncertainty quantification; that is, the linearised Laplace method. Alas, the computational cost associated with Bayesian linear models constrains this method's application to small networks, small output spaces and small datasets. We address this limitation by introducing a scalable sample-based Bayesian inference method for conjugate Gaussian multi-output linear models, together with a matching method for hyperparameter (regularisation) selection. Furthermore, we use a classic feature normalisation method (the g-prior) to resolve a previously highlighted pathology of the linearised Laplace method. Together, these contributions allow us to perform linearised neural network inference with ResNet-18 on CIFAR100 (11M parameters, 100 outputs x 50k datapoints), with ResNet-50 on Imagenet (50M parameters, 1000 outputs x 1.2M datapoints) and with a U-Net on a high-resolution tomographic reconstruction task (2M parameters, 251k output~dimensions).

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 41 of its 77 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

41 of 77 references.

References · 41

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  5. 2017Deep Image Priorcited by 1,097
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  9. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  10. 2013Gaussian Processes for Big Datacited by 377
  11. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  12. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
29 more
  1. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  2. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  3. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  4. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  5. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  6. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 125
  7. 2021Epistemic Neural Networkscited by 120
  8. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  9. 2018Uncertainty in Neural Networks: Approximately Bayesian Ensemblingcited by 94
  10. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  11. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
  12. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  13. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  14. 2020Computed Tomography Reconstruction Using Deep Image Prior and Learned Reconstruction Methodscited by 56
  15. 2020Depth Uncertainty in Neural Networkscited by 56
  16. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  17. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  18. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  19. 2013Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regressioncited by 32
  20. 2021Fast Adaptation with Linearized Neural Networkscited by 32
  21. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  22. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  23. 2021Dual Parameterization of Sparse Variational Gaussian Processescited by 19
  24. 2020Uncertainty Estimation in Medical Image Denoising with Bayesian Deep Image Priorcited by 18
  25. 2019Disentangling and Learning Robust Representations with Natural Clusteringcited by 10
  26. 2022Evaluating High-Order Predictive Distributions in Deep Learningcited by 10
  27. 2021Anti-Concentrated Confidence Bonuses for Scalable Explorationcited by 7
  28. 2022Bayesian Experimental Design for Computed Tomography with the Linearised Deep Image Priorcited by 7
  29. 2021Interoperability of statistical models in pandemic preparedness: principles and realitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 2; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  2. 2023Sampling from Gaussian Process Posteriors using Stochastic Gradient Descentcited by 27
  3. 2023Variational Linearized Laplace Approximation for Bayesian Deep Learningcited by 17
  4. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  5. 2023Stochastic Gradient Descent for Gaussian Processes Done Rightcited by 16
  6. 2023Exploration via linearly perturbed loss minimisationcited by 13
  7. 2024FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learningcited by 11
  8. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  9. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  10. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  11. 2024Partially Stochastic Infinitely Deep Bayesian Neural Networkscited by 6
  12. 2024Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processescited by 5
6 more
  1. 2025Diffusion Active Learning: Towards Data-Driven Experimental Design in Computed Tomographycited by 4
  2. 2024Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscited by 3
  3. 2023Online Laplace Model Selection Revisitedcited by 2
  4. 2024Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learningcited by 2
  5. 2024Warm Start Marginal Likelihood Optimisation for Iterative Gaussian Processescited by 1
  6. 2026Is the Last Layer Sufficient for Uncertainty Quantification?cited by 1