mlsanityData to 7 Oct 2026

Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projections

arXiv 2410.16901 ↗First posted 22 Oct 2024cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Bayesian deep learning all too often underfits so that the Bayesian prediction is less accurate than a simple point estimate. Uncertainty quantification then comes at the cost of accuracy. For linearized models, the null space of the generalized Gauss-Newton matrix corresponds to parameters that preserve the training predictions of the point estimate. We propose to build Bayesian approximations in this null space, thereby guaranteeing that the Bayesian predictive does not underfit. We suggest a matrix-free algorithm for projecting onto this null space, which scales linearly with the number of parameters and quadratically with the number of output dimensions. We further propose an approximation that only scales linearly with parameters to make the method applicable to generative models. An extensive empirical evaluation shows that the approach scales to large models, including vision transformers with 28 million parameters.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 70 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

36 of 70 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  7. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  10. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  11. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  12. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
24 more
  1. 2022Visual Attention Networkcited by 210
  2. 2019Practical Deep Learning with Bayesian Principlescited by 158
  3. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  4. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  5. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  6. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  7. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  8. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  9. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  10. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  11. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  12. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
  13. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
  14. 2019BackPACK: Packing more into backpropcited by 77
  15. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  16. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  17. 2021Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networkscited by 32
  18. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 27
  19. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  20. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  21. 2021Being a Bit Frequentist Improves Bayesian Neural Networkscited by 15
  22. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  23. 2023Large-Scale Gaussian Processes via Alternating Projectioncited by 11
  24. 2024Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihoodcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  2. 2025VIKING: Deep variational inference with stochastic projectionscited by 3
  3. 2026Laplace Approximation for Bayesian Tensor Network Kernel Machinescited by 1