On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors
Not on the map. Why?
Abstract (arXiv)
Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 51 of its 88 references that are in the map. First posted on arXiv 23 Mar 2026.
Builds on
51 of 88 references.
References · 51
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
- 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
- 2014New insights and perspectives on the natural gradient methodcited by 435
- 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
- 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
- 2018Invariant and Equivariant Graph Networkscited by 339
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
- 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
- 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
38 more
- 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
- 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
- 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
- 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
- 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
- 2019Subspace Inference for Bayesian Deep Learningcited by 94
- 2019Reverse-Engineering Deep ReLU Networkscited by 81
- 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
- 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
- 2013Understanding predictive information criteria for Bayesian modelscited by 51
- 2019On Connected Sublevel Sets in Deep Learningcited by 51
- 2019Challenges in Markov chain Monte Carlo for Bayesian neural networkscited by 43
- 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
- 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
- 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
- 2023Symmetry Induces Structure and Constraint of Learningcited by 25
- 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
- 2024BlackJAX: Composable Bayesian inference in JAXcited by 24
- 2022Wide Mean-Field Bayesian Neural Networks Ignore the Datacited by 22
- 2018Overpruning in Variational Bayesian Neural Networkscited by 21
- 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
- 2023Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetrycited by 20
- 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
- 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
- 2024Scalable Bayesian Learning with posteriorscited by 13
- 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 13
- 2023A Symmetry-Aware Exploration of Bayesian Neural Network Posteriorscited by 12
- 2024Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamicscited by 12
- 2023Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrizationcited by 11
- 2024Exploring the loss landscape of regularized neural networks via convex dualitycited by 11
- 2024Remove Symmetries to Control Model Expressivity and Improve Optimizationcited by 8
- 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
- 2023Fluctuation without dissipation: Microcanonical Langevin Monte Carlocited by 6
- 2024Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networkscited by 6
- 2025Paths and Ambient Spaces in Neural Loss Landscapescited by 6
- 2026Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?cited by 6
- 2020Bayesian neural networks and dimensionality reductioncited by 4
- 2022Approximate blocked Gibbs sampling for Bayesian neural networkscited by 3
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.