mlsanityData to 14 Oct 2026

Reparameterization invariance in approximate Bayesian inference

arXiv 2406.03334 ↗First posted 5 Jun 2024cs.LG stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Current approximate posteriors in Bayesian neural networks (BNNs) exhibit a crucial limitation: they fail to maintain invariance under reparameterization, i.e. BNNs assign different posterior densities to different parametrizations of identical functions. This creates a fundamental flaw in the application of Bayesian principles as it breaks the correspondence between uncertainty over the parameters with uncertainty over the parametrized function. In this paper, we investigate this issue in the context of the increasingly popular linearized Laplace approximation. Specifically, it has been observed that linearized predictives alleviate the common underfitting problems of the Laplace approximation. We develop a new geometric view of reparametrizations from which we explain the success of linearization. Moreover, we demonstrate that these reparameterization invariance properties can be extended to the original neural network predictive using a Riemannian diffusion process giving a straightforward algorithm for approximate posterior sampling, which empirically improves posterior fit.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

33 of 56 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  6. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  8. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  9. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  10. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  11. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  12. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
21 more
  1. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  2. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  3. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  4. 2019Subspace Inference for Bayesian Deep Learningcited by 95
  5. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  6. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 88
  7. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
  8. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  9. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  10. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  11. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  12. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  13. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 28
  14. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  15. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  16. 2023Riemannian Laplace approximations for Bayesian neural networkscited by 15
  17. 2022Laplacian Autoencoders for Learning Stochastic Representationscited by 10
  18. 2023Bayesian Metric Learning for Uncertainty Quantification in Image Retrievalcited by 10
  19. 2019A Reparameterization-Invariant Flatness Measure for Deep Neural Networkscited by 7
  20. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  21. 2023Riemannian Laplace Approximation with the Fisher Metriccited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 5.

17 papers.

Sort:

By area:

  1. 2025laplax -- Laplace Approximations with JAXcited by 5
  2. 2026Richer Bayesian Last Layers with Subsampled NTK Featurescited by 4
  3. 2024Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Samplingcited by 3
  4. 2024Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscited by 3
  5. 2025Identifying Metric Structures of Deep Latent Variable Modelscited by 3
  6. 2025VIKING: Deep variational inference with stochastic projectionscited by 3
  7. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 3
  8. 2024Debiasing Mini-Batch Quadratics for Applications in Deep Learningcited by 2
  9. 2025Geometric Gaussian Approximations of Probability Distributionscited by 2
  10. 2026On the Epistemic Uncertainty of Overparametrized Neural Networkscited by 2
  11. 2024ELBOing Stein: Variational Bayes with Stein Mixture Inferencecited by 1
  12. 2025Tubular Riemannian Laplace Approximations for Bayesian Neural Networkscited by 1
5 more
  1. 2026Reducing Memorisation in Generative Models via Riemannian Bayesian Inferencecited by 1
  2. 2024Robust Domain Generalisation with Causal Invariant Bayesian Neural Networkscited by 0
  3. 2025On Local Posterior Structure in Deep Ensemblescited by 0
  4. 2025POEMS: Product of Experts for Interpretable Multi-omic Integration using Sparse Decodingcited by 0
  5. 2026Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanicscited by 0